-
Joao Gramacho authored
EXPECTED Background ---------- Binlog_sender writes events from the binary log to a packet buffer, then sends the packet to the slave. The packet buffer is kept between events. Before an event is written to the buffer, if the event is bigger than the current allocated size of the buffer, the buffer is reallocated. To keep reallocations few, it grows to at least twice the size. To prevent that the buffer stays big forever just because there was one big event, the buffer shrinks when events are small again. To prevent that the buffer grows and shrinks often, it shrinks to half the size if 100 events in sequence are smaller than half the buffer size. Problems -------- P1) The buffer shrink procedure is not working as expected. The buffer is not shrinking to half the size if 100 events in sequence are smaller than half the buffer size. P2) The buffer might be reallocated by Log_event::read_log_event(). As the packet buffer is pre-allocated before loading the event, the buffer should never be reallocated by Log_event::read_log_event(). Analysis P1) Due to a typo in the code, the buffer shrinks to 4096 bytes when 100 events in sequence are smaller than 4096 bytes. Also, if one event in 100 is at least 4097 bytes big, the buffer will never shrink. P2) The Binlog_sender uses String::append(IO_CACHE, size) to load the body part of the event into the (packet) buffer. The issue happens when the buffer has BS bytes allocated and the event size is ES = BS - 1, BS is a value aligned by 8 bytes. So, when an event of size ES = BS - 1 is going to be loaded to a buffer with BS bytes of allocated size (BS = ES + 1), the Binlog_sender expects no changes in the buffer size (or pointers) as 1 (OK) + ES <= BS. The packet buffer is always initialized with a 0 to sign a OK packet, making packet->m_length= 1. Then, Log_event::read_log_event appends LOG_EVENT_MINIMAL_HEADER_LEN (19 bytes), making packet->m_length= 20. Finally, Log_event::read_log_event loads the rest of the event content into the buffer, calling String::append(IO_CACHE, ES - 19). String::append is then calling String::mem_realloc(m_length + append_size), or (20 + ES - 19 = 1 + ES) doing the math. The current buffer is enough to hold the event (BS = ES + 1), but there is an "+ 1" used at the new desired buffer "len" calculation (to store a new \0 for the string) using ALIGN_SIZE that will make the buffer to grow 8 more bytes, producing the packet buffer re-allocation. Fix --- Separated growth and shrink into distinct functions by creating Binlog_sender::calc_shrink_buffer_size and renaming Binlog_sender::calc_buffer_size as Binlog_sender::calc_grow_buffer_size. Changed Log_event::read_log_event(IO_CACHE*, String*, ...) making it to read the event content directly from the I/O cache to the packet buffer instead of calling String::append(). Added a test case to verify the buffer growth and shrink.
Joao Gramacho authoredEXPECTED Background ---------- Binlog_sender writes events from the binary log to a packet buffer, then sends the packet to the slave. The packet buffer is kept between events. Before an event is written to the buffer, if the event is bigger than the current allocated size of the buffer, the buffer is reallocated. To keep reallocations few, it grows to at least twice the size. To prevent that the buffer stays big forever just because there was one big event, the buffer shrinks when events are small again. To prevent that the buffer grows and shrinks often, it shrinks to half the size if 100 events in sequence are smaller than half the buffer size. Problems -------- P1) The buffer shrink procedure is not working as expected. The buffer is not shrinking to half the size if 100 events in sequence are smaller than half the buffer size. P2) The buffer might be reallocated by Log_event::read_log_event(). As the packet buffer is pre-allocated before loading the event, the buffer should never be reallocated by Log_event::read_log_event(). Analysis P1) Due to a typo in the code, the buffer shrinks to 4096 bytes when 100 events in sequence are smaller than 4096 bytes. Also, if one event in 100 is at least 4097 bytes big, the buffer will never shrink. P2) The Binlog_sender uses String::append(IO_CACHE, size) to load the body part of the event into the (packet) buffer. The issue happens when the buffer has BS bytes allocated and the event size is ES = BS - 1, BS is a value aligned by 8 bytes. So, when an event of size ES = BS - 1 is going to be loaded to a buffer with BS bytes of allocated size (BS = ES + 1), the Binlog_sender expects no changes in the buffer size (or pointers) as 1 (OK) + ES <= BS. The packet buffer is always initialized with a 0 to sign a OK packet, making packet->m_length= 1. Then, Log_event::read_log_event appends LOG_EVENT_MINIMAL_HEADER_LEN (19 bytes), making packet->m_length= 20. Finally, Log_event::read_log_event loads the rest of the event content into the buffer, calling String::append(IO_CACHE, ES - 19). String::append is then calling String::mem_realloc(m_length + append_size), or (20 + ES - 19 = 1 + ES) doing the math. The current buffer is enough to hold the event (BS = ES + 1), but there is an "+ 1" used at the new desired buffer "len" calculation (to store a new \0 for the string) using ALIGN_SIZE that will make the buffer to grow 8 more bytes, producing the packet buffer re-allocation. Fix --- Separated growth and shrink into distinct functions by creating Binlog_sender::calc_shrink_buffer_size and renaming Binlog_sender::calc_buffer_size as Binlog_sender::calc_grow_buffer_size. Changed Log_event::read_log_event(IO_CACHE*, String*, ...) making it to read the event content directly from the I/O cache to the packet buffer instead of calling String::append(). Added a test case to verify the buffer growth and shrink.
Loading