You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2026/10/07 18:37:00 UTC
[jira] [Commented] (IMPALA-15358) Cap writers to Kudu partitions
[ https://issues.apache.org/jira/browse/IMPALA-15358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124650#comment-18124650 ]
ASF subversion and git services commented on IMPALA-15358:
----------------------------------------------------------
Commit 3ed95af420c104d5a61bca67c5a2747603613f28 in impala's branch refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3ed95af42 ]
IMPALA-15358: Cap Kudu DML writers to partition count
For INSERT/UPSERT into a partitioned Kudu table, the planner inserts a
KUDU-partitioned exchange ahead of the KuduTableSink. At runtime,
KrpcDataStreamSender routes each row to channel (partition_index %
num_channels), where partition_index is the actual Kudu tablet index
(bounded by the table's tablet count). The writer fragment's instance
count, however, was sized like any other fragment (mt_dop / cost-based
scaling) with no awareness of the target table's partitioning. Whenever
the scheduler assigned more writer instances than the table has tablets,
instances index >= tablet count never received rows, causing work skew.
Cap the writer fragment's instance count at the Kudu table's partition
count via an RPC, reusing the existing MAX_FS_WRITERS mechanism
(DmlStatementBase.maxTableSinks_). KuduTableSink now honors the same cap
via PlanFragment.getNumNodes()/getNumInstances().
Updates Kudu support for MAX_FS_WRITERS and CPC to match HDFS.
Also caps CTAS based on an estimate of the number of tablets.
Adds planner and end-to-end tests. Planner tests with CPC use
approximate segment-costs as they're susceptible to small differences in
statistics that can vary across environments.
Change-Id: Ia3644fb245ceac72e91ed70c8c0392f65df75fde
Assisted-by: Claude Sonnet 4.5, 5.5 (GitHub Copilot)
Reviewed-on: http://gerrit.cloudera.org:8080/24852
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
> Cap writers to Kudu partitions
> ------------------------------
>
> Key: IMPALA-15358
> URL: https://issues.apache.org/jira/browse/IMPALA-15358
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Michael Smith
> Assignee: Michael Smith
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> For INSERT/UPSERT/UPDATE/DELETE into a partitioned Kudu table, the planner inserts a KUDU-partitioned exchange ahead of the KuduTableSink. At runtime, KrpcDataStreamSender routes each row to channel (partition_index % num_channels), where partition_index is the actual Kudu tablet index (bounded by the table's tablet count). The writer fragment's instance count, however, is sized like any other fragment (mt_dop / cost-based scaling) with no awareness of the target table's partitioning. Whenever the scheduler assigns more writer instances than the table has tablets, every instance index >= tablet count never receives rows, causing severe work skew.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org