filesort() initializes space for
MIN(Estimate rows in table, max rows that can fit in sort buffer) rows
before doing the sort. For queries that have a "where" clause, the
number of rows estimated by estimate_rows_upper_bound can be much
higher than rows that match the where clause.

The initialization is done via make_char_array() in filesort.cc and is
very expensive when the estimate is off.

Solution: lazy-init of the pointers in the sort buffer.