You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Balazs Hevele (Jira)" <ji...@apache.org> on 2026/09/16 16:57:00 UTC
[jira] [Commented] (IMPALA-15181) Grouping aggregations are not making use of small strings
[ https://issues.apache.org/jira/browse/IMPALA-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116124#comment-18116124 ]
Balazs Hevele commented on IMPALA-15181:
----------------------------------------
Gerrit patch: https://gerrit.cloudera.org/#/c/24848/
> Grouping aggregations are not making use of small strings
> ---------------------------------------------------------
>
> Key: IMPALA-15181
> URL: https://issues.apache.org/jira/browse/IMPALA-15181
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Balazs Hevele
> Priority: Major
>
> TPC-H Q67 has a streaming pre-agg with several grouping aggregations. Some of the columns used for the keys are small strings that are eligible for the small string optimization (e.g. item's i_category and i_class). I added some debugging code to determine how much memory the small strings optimization was saving for grouping aggregations.
> {noformat}
> int GroupingAggregator::GroupingExprsVarlenSize() {
> int varlen_size = 0;
> // TODO: The hash table could compute this as it hashes.
> for (int expr_idx : string_grouping_exprs_) {
> StringValue* sv = reinterpret_cast<StringValue*>(ht_ctx_->ExprValue(expr_idx));
> if (ht_ctx_->ExprValueNull(expr_idx)) continue;
> if (sv->IsSmall()) {
> num_small_strings_++;
> small_strings_size_ += sv->Len();
> } else if (sv->Len() <= 11) {
> num_potential_small_strings_++;
> potential_small_strings_size_ += sv->Len();
> }
> varlen_size += sv->Len();
> }
> varlen_data_size_ += varlen_size;
> return varlen_size;
> }
> ...
> Status GroupingAggregator::GetNext(RuntimeState* state, RowBatch* row_batch, bool* eos) {
> RETURN_IF_ERROR(QueryMaintenance(state));
> if (!partition_eos_) {
> RETURN_IF_ERROR(GetRowsFromPartition(state, row_batch));
> }
> *eos = partition_eos_;
> if (*eos) {
> LOG(INFO) << "GroupingAggregator reached eos. Varlen data size: "
> << varlen_data_size_ << "\n"
> << "\tNum small strings: " << num_small_strings_
> << " Small strings size: " << small_strings_size_ << "\n"
> << "\tPotential small strings: " << num_potential_small_strings_
> << " potential small strings size: " << potential_small_strings_size_
> << "\tCopyGroupingValues small strings: " << copygroupingvalues_small_strings_;
> }
> return Status::OK();
> }{noformat}
> For TPC-DS Q67 on scale factor 5, this produced:
> {noformat}
> I20260714 15:37:53.094179 710844 grouping-aggregator.cc:274] 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data size: 55366901
> Num small strings: 0 Small strings size: 0
> Potential small strings: 1642739 potential small strings size: 10722894 CopyGroupingValues small strings: 0
> I20260714 15:37:53.445653 710844 grouping-aggregator.cc:274] 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data size: 14319844
> Num small strings: 0 Small strings size: 0
> Potential small strings: 563651 potential small strings size: 3678131 CopyGroupingValues small strings: 0
> I20260714 15:37:53.565832 710844 grouping-aggregator.cc:274] 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data size: 5262020
> Num small strings: 0 Small strings size: 0
> Potential small strings: 207104 potential small strings size: 1351759 CopyGroupingValues small strings: 0
> I20260714 15:37:53.596243 710844 grouping-aggregator.cc:274] 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data size: 1322171
> Num small strings: 0 Small strings size: 0
> Potential small strings: 52036 potential small strings size: 339642 CopyGroupingValues small strings: 0
> I20260714 15:37:53.624203 710844 grouping-aggregator.cc:274] 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data size: 1322171
> Num small strings: 0 Small strings size: 0
> Potential small strings: 52036 potential small strings size: 339642 CopyGroupingValues small strings: 0
> I20260714 15:37:53.624967 710844 grouping-aggregator.cc:274] 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data size: 21408
> Num small strings: 0 Small strings size: 0
> Potential small strings: 1260 potential small strings size: 9688 CopyGroupingValues small strings: 0
> I20260714 15:37:53.625154 710844 grouping-aggregator.cc:274] 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data size: 1670
> Num small strings: 0 Small strings size: 0
> Potential small strings: 217 potential small strings size: 1454 CopyGroupingValues small strings: 0
> I20260714 15:37:53.625212 710844 grouping-aggregator.cc:274] 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data size: 59
> Num small strings: 0 Small strings size: 0
> Potential small strings: 10 potential small strings size: 59 CopyGroupingValues small strings: 0
> I20260714 15:37:53.625252 710844 grouping-aggregator.cc:274] 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data size: 0
> Num small strings: 0 Small strings size: 0
> Potential small strings: 0 potential small strings size: 0 CopyGroupingValues small strings: 0{noformat}
> Based on this, there are strings with length <= 11 that should be eligible, but small strings aren't being used by the grouping aggregator. I ran TPC-H / TPC-DS queries and didn't see any with small strings. Fixing this could reduce memory use (10MB for aggregator 0?) and avoid a read for the variable length buffer for the small strings (which is on a hot path).
> It's hard to notice this type of issue. I'm not sure how we could test this more methodically, but it's hard to tell if there are similar issues with other locations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org