Skip to content
  • Joao Gramacho's avatar
    123be400
    BUG#24643036 BINLOG_SENDER DOES NOT REDUCE SIZE OF SEND BUFFER AS · 123be400
    Joao Gramacho authored
                 EXPECTED
    
    Background
    ----------
    
    Binlog_sender writes events from the binary log to a packet buffer, then
    sends the packet to the slave. The packet buffer is kept between events.
    
    Before an event is written to the buffer, if the event is bigger than
    the current allocated size of the buffer, the buffer is reallocated. To
    keep reallocations few, it grows to at least twice the size.
    
    To prevent that the buffer stays big forever just because there was one
    big event, the buffer shrinks when events are small again.
    
    To prevent that the buffer grows and shrinks often, it shrinks to half
    the size if 100 events in sequence are smaller than half the buffer
    size.
    
    Problems
    --------
    
    P1) The buffer shrink procedure is not working as expected.
    
    The buffer is not shrinking to half the size if 100 events in sequence
    are smaller than half the buffer size.
    
    P2) The buffer might be reallocated by Log_event::read_log_event().
    
    As the packet buffer is pre-allocated before loading the event, the
    buffer should never be reallocated by Log_event::read_log_event().
    
    Analysis
    
    P1)
    
    Due to a typo in the code, the buffer shrinks to 4096 bytes when 100
    events in sequence are smaller than 4096 bytes.
    
    Also, if one event in 100 is at least 4097 bytes big, the buffer will
    never shrink.
    
    P2)
    
    The Binlog_sender uses String::append(IO_CACHE, size) to load the body
    part of the event into the (packet) buffer.
    
    The issue happens when the buffer has BS bytes allocated and the event
    size is ES = BS - 1, BS is a value aligned by 8 bytes.
    
    So, when an event of size ES = BS - 1 is going to be loaded to a buffer
    with BS bytes of allocated size (BS = ES + 1), the Binlog_sender expects
    no changes in the buffer size (or pointers) as 1 (OK) + ES <= BS.
    
    The packet buffer is always initialized with a 0 to sign a OK packet,
    making packet->m_length= 1.
    
    Then, Log_event::read_log_event appends LOG_EVENT_MINIMAL_HEADER_LEN (19
    bytes), making packet->m_length= 20.
    
    Finally, Log_event::read_log_event loads the rest of the event content
    into the buffer, calling String::append(IO_CACHE, ES - 19).
    
    String::append is then calling String::mem_realloc(m_length +
    append_size), or (20 + ES - 19 = 1 + ES) doing the math. The current
    buffer is enough to hold the event (BS = ES + 1), but there is an "+ 1"
    used at the new desired buffer "len" calculation (to store a new \0 for
    the string) using ALIGN_SIZE that will make the buffer to grow 8 more
    bytes, producing the packet buffer re-allocation.
    
    Fix
    ---
    
    Separated growth and shrink into distinct functions by creating
    Binlog_sender::calc_shrink_buffer_size and renaming
    Binlog_sender::calc_buffer_size as Binlog_sender::calc_grow_buffer_size.
    
    Changed Log_event::read_log_event(IO_CACHE*, String*, ...) making it to
    read the event content directly from the I/O cache to the packet buffer
    instead of calling String::append().
    
    Added a test case to verify the buffer growth and shrink.
    123be400
    BUG#24643036 BINLOG_SENDER DOES NOT REDUCE SIZE OF SEND BUFFER AS
    Joao Gramacho authored
                 EXPECTED
    
    Background
    ----------
    
    Binlog_sender writes events from the binary log to a packet buffer, then
    sends the packet to the slave. The packet buffer is kept between events.
    
    Before an event is written to the buffer, if the event is bigger than
    the current allocated size of the buffer, the buffer is reallocated. To
    keep reallocations few, it grows to at least twice the size.
    
    To prevent that the buffer stays big forever just because there was one
    big event, the buffer shrinks when events are small again.
    
    To prevent that the buffer grows and shrinks often, it shrinks to half
    the size if 100 events in sequence are smaller than half the buffer
    size.
    
    Problems
    --------
    
    P1) The buffer shrink procedure is not working as expected.
    
    The buffer is not shrinking to half the size if 100 events in sequence
    are smaller than half the buffer size.
    
    P2) The buffer might be reallocated by Log_event::read_log_event().
    
    As the packet buffer is pre-allocated before loading the event, the
    buffer should never be reallocated by Log_event::read_log_event().
    
    Analysis
    
    P1)
    
    Due to a typo in the code, the buffer shrinks to 4096 bytes when 100
    events in sequence are smaller than 4096 bytes.
    
    Also, if one event in 100 is at least 4097 bytes big, the buffer will
    never shrink.
    
    P2)
    
    The Binlog_sender uses String::append(IO_CACHE, size) to load the body
    part of the event into the (packet) buffer.
    
    The issue happens when the buffer has BS bytes allocated and the event
    size is ES = BS - 1, BS is a value aligned by 8 bytes.
    
    So, when an event of size ES = BS - 1 is going to be loaded to a buffer
    with BS bytes of allocated size (BS = ES + 1), the Binlog_sender expects
    no changes in the buffer size (or pointers) as 1 (OK) + ES <= BS.
    
    The packet buffer is always initialized with a 0 to sign a OK packet,
    making packet->m_length= 1.
    
    Then, Log_event::read_log_event appends LOG_EVENT_MINIMAL_HEADER_LEN (19
    bytes), making packet->m_length= 20.
    
    Finally, Log_event::read_log_event loads the rest of the event content
    into the buffer, calling String::append(IO_CACHE, ES - 19).
    
    String::append is then calling String::mem_realloc(m_length +
    append_size), or (20 + ES - 19 = 1 + ES) doing the math. The current
    buffer is enough to hold the event (BS = ES + 1), but there is an "+ 1"
    used at the new desired buffer "len" calculation (to store a new \0 for
    the string) using ALIGN_SIZE that will make the buffer to grow 8 more
    bytes, producing the packet buffer re-allocation.
    
    Fix
    ---
    
    Separated growth and shrink into distinct functions by creating
    Binlog_sender::calc_shrink_buffer_size and renaming
    Binlog_sender::calc_buffer_size as Binlog_sender::calc_grow_buffer_size.
    
    Changed Log_event::read_log_event(IO_CACHE*, String*, ...) making it to
    read the event content directly from the I/O cache to the packet buffer
    instead of calling String::append().
    
    Added a test case to verify the buffer growth and shrink.
Loading