Skip to content
  • Steinar H. Gunderson's avatar
    71464a1e
    WL #11590: More flexible filesort [patch 3/10, incremental buffer] · 71464a1e
    Steinar H. Gunderson authored
    Instead of allocating one big filesort buffer up-front, start with
    a small buffer, only increasing it (exponentially) once we actually get more
    data. If we actually need all of the buffer, we do somewhat more mallocs
    than in the allocate-once case, but on Linux, the mallocs disappear
    entirely in the rest of the work. (On Windows, we do see a regression
    in heavily concurrent sort benchmarks, but that will be fixed in a future
    worklog.)
    
    For the case where we have a large buffer but don't use it (e.g. 256 kB
    of data with 32 MB large buffer), we do save a fair amount of CPU time (on the
    order of 7–8%, although of course this will vary with the exact circumstances).
    Also, of course, we save the RAM we don't use.
    
    Note that we never move data between the sort buffers -- once rows are written
    into a buffer, they never move. This means we need separate storage for the
    record pointers, which means we also need some care to make sure we don't
    overrun the sort buffer budget too much with the hidden cost of storing the
    pointers.
    
    Change-Id: I0599bc10947aec95ae80a3dbbf04e81d2017cfb0
    71464a1e
    WL #11590: More flexible filesort [patch 3/10, incremental buffer]
    Steinar H. Gunderson authored
    Instead of allocating one big filesort buffer up-front, start with
    a small buffer, only increasing it (exponentially) once we actually get more
    data. If we actually need all of the buffer, we do somewhat more mallocs
    than in the allocate-once case, but on Linux, the mallocs disappear
    entirely in the rest of the work. (On Windows, we do see a regression
    in heavily concurrent sort benchmarks, but that will be fixed in a future
    worklog.)
    
    For the case where we have a large buffer but don't use it (e.g. 256 kB
    of data with 32 MB large buffer), we do save a fair amount of CPU time (on the
    order of 7–8%, although of course this will vary with the exact circumstances).
    Also, of course, we save the RAM we don't use.
    
    Note that we never move data between the sort buffers -- once rows are written
    into a buffer, they never move. This means we need separate storage for the
    record pointers, which means we also need some care to make sure we don't
    overrun the sort buffer budget too much with the hidden cost of storing the
    pointers.
    
    Change-Id: I0599bc10947aec95ae80a3dbbf04e81d2017cfb0
Loading