Skip to content
  • Oystein Grovlen's avatar
    809a2e31
    Bug#11766758: MATERIALIZATION=ON/OFF LEADS TO DIFFERENT RESULT SET · 809a2e31
    Oystein Grovlen authored
    WHEN USING IN & VAR_POP
    
    This bug is caused by the fact that number of decimals in VAR_POP
    result, was calculated as the number of decimals in the argument 
    (here int => 0) plus 4.
    
    Hence, in the example in the bug report, 3.5556 is stored in t2.X.
    
    When materialization is off, result of inner query is compared using
    Arg_comparator::compare_real_fixed() which takes number of decimals into
    account.
    
    When inner query is materialized, no truncation happens and the materialized
    column contains 3.555555555...
    The IN-expression is evaluated by doing hash-lookup into the materialized
    table, and since no record matches 3.5556, the result set is empty.
    
    According to the SQL standard the result of VAR_POP should be a 
    floating point number. Hence, the result should not be truncated to 
    a certain number of decimals.  This patch changes VAR_POP() and similar
    functions (VAR_SAMP(), VARIANCE(), STD(), STDDEV_POP(), STDDEV_SAMP(), and
    STDDEV())
    
    By changing this, results for these functions will never be truncated
    Hence, materialization should no longer give a different value from 
    what is normally returned to users.
    809a2e31
    Bug#11766758: MATERIALIZATION=ON/OFF LEADS TO DIFFERENT RESULT SET
    Oystein Grovlen authored
    WHEN USING IN & VAR_POP
    
    This bug is caused by the fact that number of decimals in VAR_POP
    result, was calculated as the number of decimals in the argument 
    (here int => 0) plus 4.
    
    Hence, in the example in the bug report, 3.5556 is stored in t2.X.
    
    When materialization is off, result of inner query is compared using
    Arg_comparator::compare_real_fixed() which takes number of decimals into
    account.
    
    When inner query is materialized, no truncation happens and the materialized
    column contains 3.555555555...
    The IN-expression is evaluated by doing hash-lookup into the materialized
    table, and since no record matches 3.5556, the result set is empty.
    
    According to the SQL standard the result of VAR_POP should be a 
    floating point number. Hence, the result should not be truncated to 
    a certain number of decimals.  This patch changes VAR_POP() and similar
    functions (VAR_SAMP(), VARIANCE(), STD(), STDDEV_POP(), STDDEV_SAMP(), and
    STDDEV())
    
    By changing this, results for these functions will never be truncated
    Hence, materialization should no longer give a different value from 
    what is normally returned to users.
Loading