Skip to content
  • Chaithra Gopalareddy's avatar
    a7571080
    Bug#17769777 : EXCESSIVE MEMORY CONSUMPTION WITH MANY AND / BETWEEN ITEMS · a7571080
    Chaithra Gopalareddy authored
    Bug#17413040 : USING MANY WHERE CONDITIONS MAKES RANGE SCAN DISABLED
    
    Problem: Currently range optimizer checks for excessive memory consumption
    in a minimalistic way by checking on the MAX_SEL_ARGS created by clone().
    This leads to two problems,
    1. As there are other ways to create SEL_ARG objects and also other ways of
    memory consumption during range analysis, current check will not suffice and
    as a result range analysis can take excessive memory
    2. If a where condition is written in such a way that only clone() is invoked
    to create SEL_ARG objects, it becomes very restrictive as in the Bug#17413040.
    In this case, range optimizer might consume memory, but the plan
    generated is much more efficient than the current plan.
    
    Solution:
    In case of Bug#17769777 range optimizer requires a lot of memory and CPU
    resources because of the conditions present in the query. Currently there is
    no way to limit the range optimizer memory usage in such cases. But given that
    range optimizer uses its own memroot to allocate memory, we are putting a
    cap on this memroot's memory consumption.
    
    To keep a check on the memory consumption, a user configurable session variable
    named "range_optimizer_max_mem_size" is introduced. A value of "0" means there is
    no limit on the memory consumption.
    
    A new member 'max_capacity' is introduced in mem_root structure to limit
    the memory consumption for a mem_root.
    A new member 'allocated_size' keeps track of the memory being allocated in mem_root
    Helper functions like
    is_mem_avaiable() - to check if the requested memory can be allocated,
    set_memroot_max_capacity() - Max capacity for this memroot,
    set_memroot_error_handling() - to report error when capacity is exceeded,
    are also added.
    
    Before allocation in memroot we check if memory is available. If yes proceed
    else report error is error_for_capacity_exceeded is set. Otherwise return NULL.
    
    All the range optimizer's allocations are now made only in range_optimizer's
    memroot. Earlier SEL_ARG and SEL_TREE allocations were not done consistently
    in memroot.
    
    Error Handling for Out of memory caused due to exceeding max_capacity:
    
    To handle such cases a new error code is added. For the memroot that has
    max_capacity set, error_for_capacity_exceeded needs to be set for alloc_root
    to report error when capacity is exceeded. Else it will return NULL.
    
    For range optimizer, as the query should not be aborted when this
    error is thrown, instead a warning should be given to user about it, a
    Internal_error_handler is introduced. This handler reduces the error to a
    warning and is given out after the query execution is complete.
    a7571080
    Bug#17769777 : EXCESSIVE MEMORY CONSUMPTION WITH MANY AND / BETWEEN ITEMS
    Chaithra Gopalareddy authored
    Bug#17413040 : USING MANY WHERE CONDITIONS MAKES RANGE SCAN DISABLED
    
    Problem: Currently range optimizer checks for excessive memory consumption
    in a minimalistic way by checking on the MAX_SEL_ARGS created by clone().
    This leads to two problems,
    1. As there are other ways to create SEL_ARG objects and also other ways of
    memory consumption during range analysis, current check will not suffice and
    as a result range analysis can take excessive memory
    2. If a where condition is written in such a way that only clone() is invoked
    to create SEL_ARG objects, it becomes very restrictive as in the Bug#17413040.
    In this case, range optimizer might consume memory, but the plan
    generated is much more efficient than the current plan.
    
    Solution:
    In case of Bug#17769777 range optimizer requires a lot of memory and CPU
    resources because of the conditions present in the query. Currently there is
    no way to limit the range optimizer memory usage in such cases. But given that
    range optimizer uses its own memroot to allocate memory, we are putting a
    cap on this memroot's memory consumption.
    
    To keep a check on the memory consumption, a user configurable session variable
    named "range_optimizer_max_mem_size" is introduced. A value of "0" means there is
    no limit on the memory consumption.
    
    A new member 'max_capacity' is introduced in mem_root structure to limit
    the memory consumption for a mem_root.
    A new member 'allocated_size' keeps track of the memory being allocated in mem_root
    Helper functions like
    is_mem_avaiable() - to check if the requested memory can be allocated,
    set_memroot_max_capacity() - Max capacity for this memroot,
    set_memroot_error_handling() - to report error when capacity is exceeded,
    are also added.
    
    Before allocation in memroot we check if memory is available. If yes proceed
    else report error is error_for_capacity_exceeded is set. Otherwise return NULL.
    
    All the range optimizer's allocations are now made only in range_optimizer's
    memroot. Earlier SEL_ARG and SEL_TREE allocations were not done consistently
    in memroot.
    
    Error Handling for Out of memory caused due to exceeding max_capacity:
    
    To handle such cases a new error code is added. For the memroot that has
    max_capacity set, error_for_capacity_exceeded needs to be set for alloc_root
    to report error when capacity is exceeded. Else it will return NULL.
    
    For range optimizer, as the query should not be aborted when this
    error is thrown, instead a warning should be given to user about it, a
    Internal_error_handler is introduced. This handler reduces the error to a
    warning and is given out after the query execution is complete.
Loading