You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Riza Suminto (Code Review)" <ge...@cloudera.org> on 2024/10/14 22:06:18 UTC

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21927


Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................

IMPALA-13445: Ignore num partition for unpartitioned writes

When cost-based planning is used, writer parallelism is limited by the
number of partitions. In the unpartitioned insert scenario, there will
be just single partitions. That leads to a single fs writer only, which
causes slow writes.

This patch fixes the issue by distinguishing between partitioned insert
and unpartitioned insert. If the insert is unpartitioned, use the
byte-based estimate fully. This patch also does minor cleanup in
DistributedPlanner.java.

Testing:
- In test_executor_groups.py, move insert tests from
  test_query_cpu_count_divisor_default into separate
  test_query_cpu_count_on_insert. Add two new insert test cases there.
- Add and pass CardinalityTest.testByteBasedNumWriters().
- Pass test_executor_groups.py.

Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
---
M be/src/service/query-state-record.cc
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M tests/custom_cluster/test_executor_groups.py
5 files changed, 159 insertions(+), 67 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/21927/1
-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21927/4//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/21927/4//COMMIT_MSG@18
PS4, Line 18: 1. If the insert is unpartitioned, use the byte-based estimate fully.
            :    Shuffling should only happen if num writers is less than num input
            :    fragment instances.
            : 2. If the insert is partitioned, try to plan at least one writer for
            :    each shuffling executor nodes, but do not exceed number of
            :    partitions.
Should byte based estimate always be used regardless of partitioned or unpartitioned inserts for cases such as LIMIT or when we can estimate the number of rows being inserted are fairly small (even if the number of rows scanned is much larger). I guess for small inserts, it's not a performance issue but maybe memory estimation issue.



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 17:58:26 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 3: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Oct 2024 20:50:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 4: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Oct 2024 21:01:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 7: Code-Review+2

(1 comment)

Thanks for fixing this, LGTM!

http://gerrit.cloudera.org:8080/#/c/21927/6/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:

http://gerrit.cloudera.org:8080/#/c/21927/6/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@457
PS6, Line 457: maxWriters = totalNumPartitions
> I'm not sure. totalNumPartitions most likely has been over estimated becaus
I see, however in a lot of cases there's only a single partition column. Since maxNumWriters limit the number of writers anyway, and with hash-based shuffling we wouldn't write too many files, we could probably allow more writers. But this can be a follow-up discussion.



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 7
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 20:10:21 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 7: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 7
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 20:29:25 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Abhishek Rawat, Zoltan Borok-Nagy, David Rorke, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21927

to look at the new patch set (#3).

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................

IMPALA-13445: Ignore num partition for unpartitioned writes

When cost-based planning is used, writer parallelism is limited by the
number of partitions. In the unpartitioned insert scenario, there will
be just single partitions. That leads to a single fs writer only, which
causes slow writes.

This patch fixes the issue by distinguishing between partitioned insert
and unpartitioned insert and cause following BEHAVIOR CHANGE if
COMPUTE_PROCESSING_COST=1:

1. If the insert is unpartitioned, use the byte-based estimate fully.
   Shuffling should only happen if num writers is less than num input
   fragment instances.
2. If the insert is partitioned, try to plan at least one writer for
   each shuffling executor nodes, but do not exceed number of
   partitions.

Both partitioned and unpartitioned insert still respect MAX_FS_WRITER
option. This patch also does minor cleanup in DistributedPlanner.java.

Testing:
- In test_executor_groups.py, move insert tests from
  test_query_cpu_count_divisor_default into separate
  test_query_cpu_count_on_insert. Add some new insert test cases there.
- Add and pass CardinalityTest.testByteBasedNumWriters().
- Pass test_executor_groups.py.

Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
---
M be/src/service/query-state-record.cc
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/ProcessingCost.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/TpcdsCpuCostPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-parquet.test
M tests/custom_cluster/test_executor_groups.py
10 files changed, 1,919 insertions(+), 110 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/21927/3
-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21927/5/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:

http://gerrit.cloudera.org:8080/#/c/21927/5/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@456
PS5, Line 456: minWriters = numInputNodes;
             :         maxWriters = numInputNodes;
In this case, byteBasedNumWriters = numInputNodes.
You can re-organize code as:
  if (totalNumPartitions == 1) {
    byteBasedNumWriters = numInputNodes;
  } else {
    long minWriters = Math.min(numInputNodes, totalNumPartitions);
    long maxWriters = totalNumPartitions;
    byteBasedNumWriters = Math.max(minWriters, byteBasedNumWriters);
    byteBasedNumWriters = Math.min(maxWriters, byteBasedNumWriters);
  }



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 17:22:55 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 8: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11053/


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 8
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 19 Oct 2024 03:06:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Abhishek Rawat, Zoltan Borok-Nagy, David Rorke, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21927

to look at the new patch set (#7).

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................

IMPALA-13445: Ignore num partition for unpartitioned writes

When cost-based planning is used, writer parallelism is limited by the
number of partitions. In the unpartitioned insert scenario, there will
be just single partitions. That leads to a single fs writer only, which
causes slow writes.

This patch fixes the issue by distinguishing between partitioned insert
and unpartitioned insert and cause following BEHAVIOR CHANGE if
COMPUTE_PROCESSING_COST=1:

1. If the insert is unpartitioned, use the byte-based estimate fully.
   Shuffling should only happen if num writers is less than num input
   fragment instances.
2. If the insert is partitioned, try to plan at least one writer for
   each shuffling executor nodes, but do not exceed number of
   partitions. However, if number of partition is 1, try force writer
   colocation with input fragment.

Both partitioned and unpartitioned insert still respect MAX_FS_WRITER
option. This patch also does minor cleanup in DistributedPlanner.java.

Testing:
- In test_executor_groups.py, move insert tests from
  test_query_cpu_count_divisor_default into separate
  test_query_cpu_count_on_insert. Add some new insert test cases there.
- Add and pass CardinalityTest.testByteBasedNumWriters().
- Add new planner tests under TpcdsCpuCostPlannerTest.
- Pass test_executor_groups.py.

Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
---
M be/src/service/query-state-record.cc
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/ProcessingCost.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/TpcdsCpuCostPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-parquet.test
M tests/custom_cluster/test_executor_groups.py
10 files changed, 2,572 insertions(+), 110 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/21927/7
-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 7
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21927/1/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:

http://gerrit.cloudera.org:8080/#/c/21927/1/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@449
PS1, Line 449: if (byteBasedNumWriters <= minWriters) {
             :         // Utilize all input nodes if possible.
             :         // This is OK beacuse individual partition most likely is less than
             :         // MIN_WRITE_BYTES.
             :        
> byteBasedNumWriters value calculated in line 443 is not used for partitione
Thanks for pointing this out. I tweak the algorithm again to seek better performance, but this is still a behavior change for insert + COMPUTE_PROCESSING_COST=1. Please have another look if it make sense or not.



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Oct 2024 04:31:33 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Abhishek Rawat, Zoltan Borok-Nagy, David Rorke, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21927

to look at the new patch set (#4).

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................

IMPALA-13445: Ignore num partition for unpartitioned writes

When cost-based planning is used, writer parallelism is limited by the
number of partitions. In the unpartitioned insert scenario, there will
be just single partitions. That leads to a single fs writer only, which
causes slow writes.

This patch fixes the issue by distinguishing between partitioned insert
and unpartitioned insert and cause following BEHAVIOR CHANGE if
COMPUTE_PROCESSING_COST=1:

1. If the insert is unpartitioned, use the byte-based estimate fully.
   Shuffling should only happen if num writers is less than num input
   fragment instances.
2. If the insert is partitioned, try to plan at least one writer for
   each shuffling executor nodes, but do not exceed number of
   partitions.

Both partitioned and unpartitioned insert still respect MAX_FS_WRITER
option. This patch also does minor cleanup in DistributedPlanner.java.

Testing:
- In test_executor_groups.py, move insert tests from
  test_query_cpu_count_divisor_default into separate
  test_query_cpu_count_on_insert. Add some new insert test cases there.
- Add and pass CardinalityTest.testByteBasedNumWriters().
- Add new planner tests under TpcdsCpuCostPlannerTest.
- Pass test_executor_groups.py.

Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
---
M be/src/service/query-state-record.cc
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/ProcessingCost.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/TpcdsCpuCostPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-parquet.test
M tests/custom_cluster/test_executor_groups.py
10 files changed, 1,924 insertions(+), 110 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/21927/4
-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21927/4//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/21927/4//COMMIT_MSG@18
PS4, Line 18: 1. If the insert is unpartitioned, use the byte-based estimate fully.
            :    Shuffling should only happen if num writers is less than num input
            :    fragment instances.
            : 2. If the insert is partitioned, try to plan at least one writer for
            :    each shuffling executor nodes, but do not exceed number of
            :    partitions.
> Should byte based estimate always be used regardless of partitioned or unpa
Scheduling less than num nodes for partitioned insert necessitate inserting exchange in-between.
It also risk opening more file handles per writers because each of them need to write to multiple partitions.

We also estimate bytes to write from output cardinality of root fragment, which can be an underestimate in certain corner cases.



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 18:46:34 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 6: Code-Review+1

(3 comments)

The code LGTM, but had a few questions.

http://gerrit.cloudera.org:8080/#/c/21927/6/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:

http://gerrit.cloudera.org:8080/#/c/21927/6/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@453
PS6, Line 453: shuflfing
nit: typo


http://gerrit.cloudera.org:8080/#/c/21927/6/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@457
PS6, Line 457: maxWriters = totalNumPartitions
Shouldn't we set maxWriters higher, e.g. totalNumPartitions * 2, to limit hash collisions and balance the load better between writers?

Base version already has this problem so it doesn't need to be solved by this patch, I just wanted to raise this question.


http://gerrit.cloudera.org:8080/#/c/21927/6/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test:

http://gerrit.cloudera.org:8080/#/c/21927/6/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test@283
PS6, Line 283: hosts=10 instances=10
Why is it 10 hosts and 10 instances when we have only a single input file?



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 19:35:47 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 8: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 8
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 21:56:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 4:

(4 comments)

Thanks for adding the extra tests!

http://gerrit.cloudera.org:8080/#/c/21927/4/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:

http://gerrit.cloudera.org:8080/#/c/21927/4/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@446
PS4, Line 446: totalNumPartitions > 0
Should this be totalNumPartitions > 1 instead? As that is very similar to the unpartitioned case.


http://gerrit.cloudera.org:8080/#/c/21927/4/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@450
PS4, Line 450: (int)
nit: this cast is unnecessary


http://gerrit.cloudera.org:8080/#/c/21927/4/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test:

http://gerrit.cloudera.org:8080/#/c/21927/4/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test@122
PS4, Line 122: ss_sold_time_sk,
             : ss_item_sk,
             : ss_customer_sk,
             : ss_cdemo_sk,
             : ss_hdemo_sk,
             : ss_addr_sk,
             : ss_store_sk,
             : ss_promo_sk,
             : ss_ticket_number,
             : ss_quantity,
             : ss_wholesale_cost,
             : ss_list_price,
             : ss_sales_price,
             : ss_ext_discount_amt,
             : ss_ext_sales_price,
             : ss_ext_wholesale_cost,
             : ss_ext_list_price,
             : ss_ext_tax,
             : ss_coupon_amt,
             : ss_net_paid,
             : ss_net_paid_inc_tax,
             : ss_net_profit,
             : ss_sold_date_sk
nit: could be *


http://gerrit.cloudera.org:8080/#/c/21927/4/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test@311
PS4, Line 311: # Partition number is 1.
             : # There should be no shuffling.
You could also have a test where many more records are written to a single partition, e.g.:

 create table store_sales_1_huge_part
 partitioned by (part_col)
 stored as iceberg as
   select a.*, 100000 as part_col
   from store_sales a, store_sales b;

For me this also generates a very high number of instances for a plan fragment:

 F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=384

It's because in setupThresholdsForExecutorGroupSets() we use setNum_cores_per_executor(Integer.MAX_VALUE), so in analyzer.getMaxParallelismPerNode() we use max_fragment_instances_per_node=128 as a limit.

Also there is a RANDOM shuffle before the writer even though the partition number is 1.



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 12:54:28 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 3:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/21927/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/21927/2//COMMIT_MSG@26
PS2, Line 26: option.
> Is it possible to add planner tests? Also for Iceberg tables?
Done through tpcds-ddl-iceberg.test. The content is exactly the same as tpcds-ddl-parquet.test. But diffing both will show different mem-estimate between the two. Otherwise, plan structure looks the same.


http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
File fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java:

http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java@245
PS2, Line 245: 
> Is it possible to have avg row size 0?
This was defensive code. I don't think it is possible.
Removing it does not seem to impact the calculation (no division by zero).


http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:

http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@448
PS2, Line 448:       // Utilize all input nodes if possible, but do not schedule more writers than
             :       // partitions.
             :       long minWriters = (int) Math.min(numInputNodes, totalNumPartitions);
             :       long maxWriters = totalNumPartitions;
             :       byteBasedNumWriters = Math.max(minWriters, byteBasedNumWriters);
             :       byteBasedNumWriters = Math.min(maxWriters, byteBasedNumWriters);
             :     }
             : 
             :     // Cap it at most maxNumWriters and at least one writer.
             :     ret
> Could it be written as the following?
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Oct 2024 20:21:42 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17224/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 17:34:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17225/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 17:54:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 5:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/21927/4/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:

http://gerrit.cloudera.org:8080/#/c/21927/4/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@446
PS4, Line 446: totalNumPartitions > 0
> Static partition INSERTs are probably not uncommon, so I think it's worth s
Done. Tested with planner test creating table customer_address_1_huge_part.


http://gerrit.cloudera.org:8080/#/c/21927/4/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@450
PS4, Line 450: Math.
> nit: this cast is unnecessary
Done


http://gerrit.cloudera.org:8080/#/c/21927/4/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test:

http://gerrit.cloudera.org:8080/#/c/21927/4/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test@122
PS4, Line 122: ---- PARALLELPLANS
             : Max Per-Host Resource Reservation: Memory=168.00MB Threads=24
             : Per-Host Resource Estimates: Memory=78.24GB
             : F01:PLAN FRAGMENT [HASH(tpcds_partitioned_parquet_snap.store_sales.ss_sold_date_sk)] hosts=10 instances=120
             : |  Per-Instance Resources: mem-estimate=6.46GB mem-reservation=6.00MB thread-reservation=1
             : |  max-parallelism=120 segment-costs=[44372981821, 97047706322]
             : WRITE TO HDFS [tpcds_partitioned_parquet_snap.store_sales_duplicate, OVERWRITE=false, PARTITION-KEYS=(ss_sold_date_sk)]
             : |  output exprs: ss_sold_time_sk, ss_item_sk, ss_customer_sk, ss_cdemo_sk, ss_hdemo_sk, ss_addr_sk, ss_store_sk, ss_promo_sk, ss_ticket_number, ss_quantity, ss_wholesale_cost, ss_list_price, ss_sales_price, ss_ext_discount_amt, ss_ext_sales_price, ss_ext_wholesale_cost, ss_ext_list_price, ss_ext_tax, ss_coupon_amt, ss_net_paid, ss_net_paid_inc_tax, ss_net_profit, ss_sold_date_sk
             : |  mem-estimate=100.00KB mem-reservation=0B thread-reservation=0 cost=97047706322
             : |
             : 02:SORT
             : |  order by: ss_sold_date_sk ASC NULLS LAST
             : |  mem-estimate=6.44GB mem-reservation=6.00MB spill-buffer=2.00MB thread-reservation=0
             : |  tuple-ids=2 row-size=96B cardinality=8.64G cost=39597689640
             : |  in pipelines: 02(GETNEXT), 00(OPEN)
             : |
             : 01:EXCHANGE [HASH(tpcds_partitioned_parquet_snap.store_sales.ss_sold_date_sk)]
             : |  mem-estimate=21.72MB mem-reservation=0B thread-reservation=0
             : |  tuple-ids=0 row-size=96B cardinality=8.64G cost=4775292181
             : |  in pipelines: 00(GETNEXT)
             : |
             : F00:PLAN FRAGMENT [RANDOM] hosts=10 instances=120
             : Per-Instance Re
> I think Planner insist that I list column names when I declare "partitioned
Done


http://gerrit.cloudera.org:8080/#/c/21927/4/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test@311
PS4, Line 311: |  |  tuple-ids=1 row-size=0B cardinality=15.00M cost=19934990
             : |  |  in pipelines: 01(GETNEXT)
> It went that high in my dev environment and I found it strange because I do
max_fragment_instances_per_node option should be the safeguard here.

  public int getMaxParallelismPerNode() {
    if (getQueryOptions().isCompute_processing_cost()) {
      return Math.max(getMinParallelismPerNode(),
          Math.min(getQueryOptions().getMax_fragment_instances_per_node(),
              getAvailableCoresPerNode()));
    } else if (getQueryOptions().getMt_dop() > 0) {
      return getQueryOptions().getMt_dop();
    } else {
      return 1;
    }
  }

Downstream, we fix it as default query option. I can look into bounding this further with information from executor group set config, but that will involve wider changes that worth its own patch.



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 17:14:47 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17229/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 9
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 19 Oct 2024 05:35:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11055/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 10
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 19 Oct 2024 05:12:10 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21927/5/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:

http://gerrit.cloudera.org:8080/#/c/21927/5/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@456
PS5, Line 456: long minWriters = Math.min(numInputNodes, totalNumPartitions);
             :         long maxWriters = totalNumP
> In this case, byteBasedNumWriters = numInputNodes.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 17:29:42 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................

IMPALA-13445: Ignore num partition for unpartitioned writes

When cost-based planning is used, writer parallelism is limited by the
number of partitions. In the unpartitioned insert scenario, there will
be just single partitions. That leads to a single fs writer only, which
causes slow writes.

This patch fixes the issue by distinguishing between partitioned insert
and unpartitioned insert and cause following BEHAVIOR CHANGE if
COMPUTE_PROCESSING_COST=1:

1. If the insert is unpartitioned, use the byte-based estimate fully.
   Shuffling should only happen if num writers is less than num input
   fragment instances.
2. If the insert is partitioned, try to plan at least one writer for
   each shuffling executor nodes, but do not exceed number of
   partitions. However, if number of partition is 1, try force writer
   colocation with input fragment.

Both partitioned and unpartitioned insert still respect MAX_FS_WRITER
option. This patch also does minor cleanup in DistributedPlanner.java.

Testing:
- In test_executor_groups.py, move insert tests from
  test_query_cpu_count_divisor_default into separate
  test_query_cpu_count_on_insert. Add some new insert test cases there.
- Add and pass CardinalityTest.testByteBasedNumWriters().
- Add new planner tests under TpcdsCpuCostPlannerTest.
- Pass test_executor_groups.py.

Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Reviewed-on: http://gerrit.cloudera.org:8080/21927
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/service/query-state-record.cc
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/ProcessingCost.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/TpcdsCpuCostPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-parquet.test
M tests/custom_cluster/test_executor_groups.py
10 files changed, 2,572 insertions(+), 110 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 11
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17227/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 7
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 20:23:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 9: Code-Review+2

There is slight planner test adjustment needed after IMPALA-13405 gets in (cardinality improvement for agg).
Reapply +2 and rerun GVO.


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 9
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 19 Oct 2024 05:11:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11053/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 8
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 21:56:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 4: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 11:27:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17213/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Oct 2024 20:41:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 4:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/21927/4/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:

http://gerrit.cloudera.org:8080/#/c/21927/4/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@446
PS4, Line 446: totalNumPartitions > 0
> Should this be totalNumPartitions > 1 instead? As that is very similar to t
This was intentional to avoid writing many files into single partition.
I don't mind making an exception, but what is the threshold? What if totalNumPartitions = 2 and there are many bytes to write?

And I believe, having more than 1 writers will not work since only one of them will receive rows from shuffling.


http://gerrit.cloudera.org:8080/#/c/21927/4/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test:

http://gerrit.cloudera.org:8080/#/c/21927/4/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test@122
PS4, Line 122: ss_sold_time_sk,
             : ss_item_sk,
             : ss_customer_sk,
             : ss_cdemo_sk,
             : ss_hdemo_sk,
             : ss_addr_sk,
             : ss_store_sk,
             : ss_promo_sk,
             : ss_ticket_number,
             : ss_quantity,
             : ss_wholesale_cost,
             : ss_list_price,
             : ss_sales_price,
             : ss_ext_discount_amt,
             : ss_ext_sales_price,
             : ss_ext_wholesale_cost,
             : ss_ext_list_price,
             : ss_ext_tax,
             : ss_coupon_amt,
             : ss_net_paid,
             : ss_net_paid_inc_tax,
             : ss_net_profit,
             : ss_sold_date_sk
> nit: could be *
I think Planner insist that I list column names when I declare "partitioned by", but I'll double check.


http://gerrit.cloudera.org:8080/#/c/21927/4/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test@311
PS4, Line 311: # Partition number is 1.
             : # There should be no shuffling.
> You could also have a test where many more records are written to a single 
It should not go that high in this planner test because tpcdsParquetCpuCostQueryOptions declare .setMax_fragment_instances_per_node(12). But I will double check.

Not sure why "100000 as part_col" is not recognized as 1 NDV, but having more than 1 writers will not work since only one of them will receive rows from shuffling.



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 14:49:31 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21927/4/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:

http://gerrit.cloudera.org:8080/#/c/21927/4/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@446
PS4, Line 446: totalNumPartitions > 0
> This was intentional to avoid writing many files into single partition.
Static partition INSERTs are probably not uncommon, so I think it's worth special casing them, then we can deal with dynamic partitioning separately.

When only a single partition is targeted it might make sense to not shuffle at all, or randomly shuffle among a set of writers.

Partitioned INSERTs are the slowest when the data volume is large, but we only have a few target partitions. For such cases probably we should invent some kind of bin packing distribution (i.e. each partition have a set of associated writers), but this is out of context of this patch.


http://gerrit.cloudera.org:8080/#/c/21927/4/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test:

http://gerrit.cloudera.org:8080/#/c/21927/4/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test@311
PS4, Line 311: # Partition number is 1.
             : # There should be no shuffling.
> It should not go that high in this planner test because tpcdsParquetCpuCost
It went that high in my dev environment and I found it strange because I don't have that many cores.

Without cost-based planning we didn't shuffle at all, which means it can regress such INSERT performance if the data volume is large.



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 15:12:14 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Abhishek Rawat, Zoltan Borok-Nagy, David Rorke, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21927

to look at the new patch set (#6).

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................

IMPALA-13445: Ignore num partition for unpartitioned writes

When cost-based planning is used, writer parallelism is limited by the
number of partitions. In the unpartitioned insert scenario, there will
be just single partitions. That leads to a single fs writer only, which
causes slow writes.

This patch fixes the issue by distinguishing between partitioned insert
and unpartitioned insert and cause following BEHAVIOR CHANGE if
COMPUTE_PROCESSING_COST=1:

1. If the insert is unpartitioned, use the byte-based estimate fully.
   Shuffling should only happen if num writers is less than num input
   fragment instances.
2. If the insert is partitioned, try to plan at least one writer for
   each shuffling executor nodes, but do not exceed number of
   partitions. However, if number of partition is 1, try force writer
   colocation with input fragment.

Both partitioned and unpartitioned insert still respect MAX_FS_WRITER
option. This patch also does minor cleanup in DistributedPlanner.java.

Testing:
- In test_executor_groups.py, move insert tests from
  test_query_cpu_count_divisor_default into separate
  test_query_cpu_count_on_insert. Add some new insert test cases there.
- Add and pass CardinalityTest.testByteBasedNumWriters().
- Add new planner tests under TpcdsCpuCostPlannerTest.
- Pass test_executor_groups.py.

Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
---
M be/src/service/query-state-record.cc
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/ProcessingCost.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/TpcdsCpuCostPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-parquet.test
M tests/custom_cluster/test_executor_groups.py
10 files changed, 2,247 insertions(+), 110 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/21927/6
-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 10: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 10
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 19 Oct 2024 10:22:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Abhishek Rawat, Zoltan Borok-Nagy, David Rorke, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21927

to look at the new patch set (#9).

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................

IMPALA-13445: Ignore num partition for unpartitioned writes

When cost-based planning is used, writer parallelism is limited by the
number of partitions. In the unpartitioned insert scenario, there will
be just single partitions. That leads to a single fs writer only, which
causes slow writes.

This patch fixes the issue by distinguishing between partitioned insert
and unpartitioned insert and cause following BEHAVIOR CHANGE if
COMPUTE_PROCESSING_COST=1:

1. If the insert is unpartitioned, use the byte-based estimate fully.
   Shuffling should only happen if num writers is less than num input
   fragment instances.
2. If the insert is partitioned, try to plan at least one writer for
   each shuffling executor nodes, but do not exceed number of
   partitions. However, if number of partition is 1, try force writer
   colocation with input fragment.

Both partitioned and unpartitioned insert still respect MAX_FS_WRITER
option. This patch also does minor cleanup in DistributedPlanner.java.

Testing:
- In test_executor_groups.py, move insert tests from
  test_query_cpu_count_divisor_default into separate
  test_query_cpu_count_on_insert. Add some new insert test cases there.
- Add and pass CardinalityTest.testByteBasedNumWriters().
- Add new planner tests under TpcdsCpuCostPlannerTest.
- Pass test_executor_groups.py.

Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
---
M be/src/service/query-state-record.cc
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/ProcessingCost.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/TpcdsCpuCostPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-parquet.test
M tests/custom_cluster/test_executor_groups.py
10 files changed, 2,572 insertions(+), 110 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/21927/9
-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 9
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
File fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java:

http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java@249
PS2, Line 249: int maxInstances = IntMath.saturatedMultiply(
             :               inputFragment.getNumNodes(), analyzer.getMaxParallelismPerNode());
> it's already calculated in line 240 as costBasedMaxWriter
Ignore it



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Oct 2024 19:20:36 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21927/1/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:

http://gerrit.cloudera.org:8080/#/c/21927/1/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@449
PS1, Line 449: if (totalNumPartitions >= maxNumWriters) {
             :         byteBasedNumWriters = maxNumWriters;
             :       } else {
             :         byteBasedNumWriters = (int) totalNumPartitions;
             :       }
byteBasedNumWriters value calculated in line 443 is not used for partitioned insert. Are you going to change the behavior for partitioned insert? Or you can move line 441 - 444 in else block.



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Oct 2024 23:53:46 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, David Rorke, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21927

to look at the new patch set (#2).

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................

IMPALA-13445: Ignore num partition for unpartitioned writes

When cost-based planning is used, writer parallelism is limited by the
number of partitions. In the unpartitioned insert scenario, there will
be just single partitions. That leads to a single fs writer only, which
causes slow writes.

This patch fixes the issue by distinguishing between partitioned insert
and unpartitioned insert and cause following BEHAVIOR CHANGE if
COMPUTE_PROCESSING_COST=1:

1. If the insert is unpartitioned, use the byte-based estimate fully.
2. If the insert is partitioned, try to plan at least one writer for
   each shuffling executor nodes, but do not exceed number of
   partitions.

Both partitioned and unpartitioned insert still respect MAX_FS_WRITER
option. This patch also does minor cleanup in DistributedPlanner.java.

Testing:
- In test_executor_groups.py, move insert tests from
  test_query_cpu_count_divisor_default into separate
  test_query_cpu_count_on_insert. Add some new insert test cases there.
- Add and pass CardinalityTest.testByteBasedNumWriters().
- Pass test_executor_groups.py.

Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
---
M be/src/service/query-state-record.cc
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M tests/custom_cluster/test_executor_groups.py
5 files changed, 231 insertions(+), 110 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/21927/2
-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 7:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/21927/4//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/21927/4//COMMIT_MSG@18
PS4, Line 18: 1. If the insert is unpartitioned, use the byte-based estimate fully.
            :    Shuffling should only happen if num writers is less than num input
            :    fragment instances.
            : 2. If the insert is partitioned, try to plan at least one writer for
            :    each shuffling executor nodes, but do not exceed number of
            :    partitions.
> Scheduling less than num nodes for partitioned insert necessitate inserting
Added store_sales_5_rows, store_sales_1_row_per_part, and store_sales_5_part_1_row_per_part.
All looks holding up pretty well.


http://gerrit.cloudera.org:8080/#/c/21927/6/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:

http://gerrit.cloudera.org:8080/#/c/21927/6/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@453
PS6, Line 453: shuffling
> nit: typo
Done


http://gerrit.cloudera.org:8080/#/c/21927/6/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@457
PS6, Line 457: maxWriters = totalNumPartitions
> Shouldn't we set maxWriters higher, e.g. totalNumPartitions * 2, to limit h
I'm not sure. totalNumPartitions most likely has been over estimated because it is a simple NDV multiplication of partition columns. So probably no need to do that.


http://gerrit.cloudera.org:8080/#/c/21927/6/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test:

http://gerrit.cloudera.org:8080/#/c/21927/6/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test@283
PS6, Line 283: costs=[7250411, 7] cp
> Why is it 10 hosts and 10 instances when we have only a single input file?
This is because we force colocation between scan and writing.
Worst case, only 1 out of 10 fragment instance is active.
Best case, file is split into 10 scan ranges and all active reading and writing.

I think this is a non-issue because the whole TpcdsCpuCostPlannerTest runs with synthetic stats from 3TB TPC-DS. So real num files in 3TB TPC-DS probably is higher or larger in size.



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 7
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 19:59:46 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11050/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 06:18:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17168/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 14 Oct 2024 22:32:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Abhishek Rawat, Zoltan Borok-Nagy, David Rorke, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21927

to look at the new patch set (#5).

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................

IMPALA-13445: Ignore num partition for unpartitioned writes

When cost-based planning is used, writer parallelism is limited by the
number of partitions. In the unpartitioned insert scenario, there will
be just single partitions. That leads to a single fs writer only, which
causes slow writes.

This patch fixes the issue by distinguishing between partitioned insert
and unpartitioned insert and cause following BEHAVIOR CHANGE if
COMPUTE_PROCESSING_COST=1:

1. If the insert is unpartitioned, use the byte-based estimate fully.
   Shuffling should only happen if num writers is less than num input
   fragment instances.
2. If the insert is partitioned, try to plan at least one writer for
   each shuffling executor nodes, but do not exceed number of
   partitions. However, if number of partition is 1, try force writer
   colocation with input fragment.

Both partitioned and unpartitioned insert still respect MAX_FS_WRITER
option. This patch also does minor cleanup in DistributedPlanner.java.

Testing:
- In test_executor_groups.py, move insert tests from
  test_query_cpu_count_divisor_default into separate
  test_query_cpu_count_on_insert. Add some new insert test cases there.
- Add and pass CardinalityTest.testByteBasedNumWriters().
- Add new planner tests under TpcdsCpuCostPlannerTest.
- Pass test_executor_groups.py.

Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
---
M be/src/service/query-state-record.cc
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/ProcessingCost.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/TpcdsCpuCostPlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/ddl.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-iceberg.test
A testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-ddl-parquet.test
M tests/custom_cluster/test_executor_groups.py
10 files changed, 2,247 insertions(+), 110 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/21927/5
-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 6: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Oct 2024 17:49:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 10: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 10
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 19 Oct 2024 05:12:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
File fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java:

http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java@249
PS2, Line 249: int maxInstances = IntMath.saturatedMultiply(
             :               inputFragment.getNumNodes(), analyzer.getMaxParallelismPerNode());
it's already calculated in line 240 as costBasedMaxWriter



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Oct 2024 19:19:21 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17214/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Oct 2024 21:25:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17205/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Oct 2024 04:52:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )

Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
......................................................................


Patch Set 2:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/21927/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/21927/2//COMMIT_MSG@26
PS2, Line 26: Testing
Is it possible to add planner tests? Also for Iceberg tables?


http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
File fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java:

http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java@245
PS2, Line 245: root.getAvgRowSize() == 0)
Is it possible to have avg row size 0?


http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:

http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@448
PS2, Line 448:       int minWriters = (int) Math.min(numInputNodes, totalNumPartitions);
             :       if (byteBasedNumWriters <= minWriters) {
             :         // Utilize all input nodes if possible.
             :         // This is OK beacuse individual partition most likely is less than
             :         // MIN_WRITE_BYTES.
             :         byteBasedNumWriters = minWriters;
             :       } else {
             :         // Cap byteBasedNumWriters at most totalNumPartitions.
             :         byteBasedNumWriters = Math.min(byteBasedNumWriters, totalNumPartitions);
             :       }
Could it be written as the following?

 long minWriters = Math.min(numInputNodes, totalNumPartitions);
 long maxWriters = totalNumPartitions;

 byteBasedNumWriters = Math.max(minWriters, byteBasedNumWriters);
 byteBasedNumWriters = Math.min(maxWriters, byteBasedNumWriters);



-- 
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Oct 2024 16:22:14 +0000
Gerrit-HasComments: Yes