Skip to content
  • Sujatha Sivakumar's avatar
    ba082ab5
    Bug#20350989: MYSQLBINLOG CAN'T DECODE EVENTS > ~1.6GB · ba082ab5
    Sujatha Sivakumar authored
    Problem:
    ========
    While converting row_data into base64 encoded format the
    row_data size conversion happens in the following manner.
    The initial row data is stored in an uint32 variable. The
    base64 encoded value needs about 4/3rds as much space of the
    raw data. Int variable is used to store the base64 bit
    encoded length. Because of this when a large event is
    received the conversion from uint32->int results in
    overflow.
    
    So instead of 2161403577 bytes (~2GB), malloc/mmap received
    a request for 18446744071575990272 (~18 ExaBytes) was issued
    due to the uint32->int->size_t conversion mess ...
    
    Analysis:
    ========
    Problem is that mysqlbinlog tries to allocate a buffer large
    enough to store a base64 encoded version of the event data
    (in sql/log_event.cc). Data length is taken from a four byte
    field in the event header, so it is limited to 4GB, which
    isn't the problem here yet. This value is then passed to
    base64_needed_encoded_length() from mysys/base64.c, and here
    disaster strikes as this function is declared as
    
    int base64_needed_encoded_length(int length_of_data)
    
    log_event.cc stores the extracted event length in uint32
    variable. This length is intern passed to the above function
    base64_needed_encoded_length which converts the size to int.
    When the event size is large and required base64 data length
    exceeds int limits this results in an overflow and results
    in out of memory error.
    
    Fix:
    ===
    Upgraded the length variable to avoid overflow in both
    encoding and decoding functions.
    
    Note: The mentioned server changes will cause following
    warnings in cluster code.
    
    storage/ndb/src/mgmapi/mgmapi.cpp:3407: warning: field width
    specifier ‘*’ expects argument of type ‘int’, but argument 3
    has type ‘size_t {aka long unsigned int}’ [-Wformat=]
    
    storage/ndb/src/mgmsrv/Config.cpp:247: warning:...
    
    The above code changes are used in cluster code to exchange
    configuration data between cluster nodes and management
    server. This code is used for both packing and unpacking the
    base64 encoded data. This data can be of max length as shown
    below which is well within int limits.
    
    if(len64 ==0 || len64 > (1024*1024)) {
      result.assfmt("Illegal config length size %d", len64);
      goto done;
    }
    
    Hence to fix these warnings for cluster code the base64
    encoded length is still typecasted to int.
    ba082ab5
    Bug#20350989: MYSQLBINLOG CAN'T DECODE EVENTS > ~1.6GB
    Sujatha Sivakumar authored
    Problem:
    ========
    While converting row_data into base64 encoded format the
    row_data size conversion happens in the following manner.
    The initial row data is stored in an uint32 variable. The
    base64 encoded value needs about 4/3rds as much space of the
    raw data. Int variable is used to store the base64 bit
    encoded length. Because of this when a large event is
    received the conversion from uint32->int results in
    overflow.
    
    So instead of 2161403577 bytes (~2GB), malloc/mmap received
    a request for 18446744071575990272 (~18 ExaBytes) was issued
    due to the uint32->int->size_t conversion mess ...
    
    Analysis:
    ========
    Problem is that mysqlbinlog tries to allocate a buffer large
    enough to store a base64 encoded version of the event data
    (in sql/log_event.cc). Data length is taken from a four byte
    field in the event header, so it is limited to 4GB, which
    isn't the problem here yet. This value is then passed to
    base64_needed_encoded_length() from mysys/base64.c, and here
    disaster strikes as this function is declared as
    
    int base64_needed_encoded_length(int length_of_data)
    
    log_event.cc stores the extracted event length in uint32
    variable. This length is intern passed to the above function
    base64_needed_encoded_length which converts the size to int.
    When the event size is large and required base64 data length
    exceeds int limits this results in an overflow and results
    in out of memory error.
    
    Fix:
    ===
    Upgraded the length variable to avoid overflow in both
    encoding and decoding functions.
    
    Note: The mentioned server changes will cause following
    warnings in cluster code.
    
    storage/ndb/src/mgmapi/mgmapi.cpp:3407: warning: field width
    specifier ‘*’ expects argument of type ‘int’, but argument 3
    has type ‘size_t {aka long unsigned int}’ [-Wformat=]
    
    storage/ndb/src/mgmsrv/Config.cpp:247: warning:...
    
    The above code changes are used in cluster code to exchange
    configuration data between cluster nodes and management
    server. This code is used for both packing and unpacking the
    base64 encoded data. This data can be of max length as shown
    below which is well within int limits.
    
    if(len64 ==0 || len64 > (1024*1024)) {
      result.assfmt("Illegal config length size %d", len64);
      goto done;
    }
    
    Hence to fix these warnings for cluster code the base64
    encoded length is still typecasted to int.
Loading