[jira] [Created] (IMPALA-15381) Reservoir sampling RNG uses excessive per-group memory

["Aleksandr Efimov (Jira)" <ji...@apache.org>]

Aleksandr Efimov created IMPALA-15381:
-----------------------------------------

             Summary: Reservoir sampling RNG uses excessive per-group memory
                 Key: IMPALA-15381
                 URL: https://issues.apache.org/jira/browse/IMPALA-15381
             Project: IMPALA
          Issue Type: Bug
          Components: Backend
            Reporter: Aleksandr Efimov


ReservoirSampleState uses boost::mt19937_64. Its RNG state is 2504 bytes, making the full state 2536 bytes. FunctionContext::Allocate() uses FreePool, which rounds this allocation to 4 KiB. Together with the initial sample buffer and allocation overhead, each sampling aggregate keeps 4624 bytes per group before the sample grows.

This affects sample(), appx_median(), and histogram(). Queries with several sampling aggregates and many groups can consume large amounts of untracked memory.

Replace the RNG with a small-state generator already available in Impala and update the planner estimate. The sampling sequence will change, so approximate results may change.

This is a regression of IMPALA-4787 introduced by IMPALA-11005.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org