-
Sujatha Sivakumar authored
Problem: ======== While converting row_data into base64 encoded format the row_data size conversion happens in the following manner. The initial row data is stored in an uint32 variable. The base64 encoded value needs about 4/3rds as much space of the raw data. Int variable is used to store the base64 bit encoded length. Because of this when a large event is received the conversion from uint32->int results in overflow. So instead of 2161403577 bytes (~2GB), malloc/mmap received a request for 18446744071575990272 (~18 ExaBytes) was issued due to the uint32->int->size_t conversion mess ... Analysis: ======== Problem is that mysqlbinlog tries to allocate a buffer large enough to store a base64 encoded version of the event data (in sql/log_event.cc). Data length is taken from a four byte field in the event header, so it is limited to 4GB, which isn't the problem here yet. This value is then passed to base64_needed_encoded_length() from mysys/base64.c, and here disaster strikes as this function is declared as int base64_needed_encoded_length(int length_of_data) log_event.cc stores the extracted event length in uint32 variable. This length is intern passed to the above function base64_needed_encoded_length which converts the size to int. When the event size is large and required base64 data length exceeds int limits this results in an overflow and results in out of memory error. Fix: === Upgraded the length variable to avoid overflow in both encoding and decoding functions. Note: The mentioned server changes will cause following warnings in cluster code. storage/ndb/src/mgmapi/mgmapi.cpp:3407: warning: field width specifier ‘*’ expects argument of type ‘int’, but argument 3 has type ‘size_t {aka long unsigned int}’ [-Wformat=] storage/ndb/src/mgmsrv/Config.cpp:247: warning:... The above code changes are used in cluster code to exchange configuration data between cluster nodes and management server. This code is used for both packing and unpacking the base64 encoded data. This data can be of max length as shown below which is well within int limits. if(len64 ==0 || len64 > (1024*1024)) { result.assfmt("Illegal config length size %d", len64); goto done; } Hence to fix these warnings for cluster code the base64 encoded length is still typecasted to int.
Sujatha Sivakumar authoredProblem: ======== While converting row_data into base64 encoded format the row_data size conversion happens in the following manner. The initial row data is stored in an uint32 variable. The base64 encoded value needs about 4/3rds as much space of the raw data. Int variable is used to store the base64 bit encoded length. Because of this when a large event is received the conversion from uint32->int results in overflow. So instead of 2161403577 bytes (~2GB), malloc/mmap received a request for 18446744071575990272 (~18 ExaBytes) was issued due to the uint32->int->size_t conversion mess ... Analysis: ======== Problem is that mysqlbinlog tries to allocate a buffer large enough to store a base64 encoded version of the event data (in sql/log_event.cc). Data length is taken from a four byte field in the event header, so it is limited to 4GB, which isn't the problem here yet. This value is then passed to base64_needed_encoded_length() from mysys/base64.c, and here disaster strikes as this function is declared as int base64_needed_encoded_length(int length_of_data) log_event.cc stores the extracted event length in uint32 variable. This length is intern passed to the above function base64_needed_encoded_length which converts the size to int. When the event size is large and required base64 data length exceeds int limits this results in an overflow and results in out of memory error. Fix: === Upgraded the length variable to avoid overflow in both encoding and decoding functions. Note: The mentioned server changes will cause following warnings in cluster code. storage/ndb/src/mgmapi/mgmapi.cpp:3407: warning: field width specifier ‘*’ expects argument of type ‘int’, but argument 3 has type ‘size_t {aka long unsigned int}’ [-Wformat=] storage/ndb/src/mgmsrv/Config.cpp:247: warning:... The above code changes are used in cluster code to exchange configuration data between cluster nodes and management server. This code is used for both packing and unpacking the base64 encoded data. This data can be of max length as shown below which is well within int limits. if(len64 ==0 || len64 > (1024*1024)) { result.assfmt("Illegal config length size %d", len64); goto done; } Hence to fix these warnings for cluster code the base64 encoded length is still typecasted to int.
Loading