You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "liuyao (Code Review)" <ge...@cloudera.org> on 2021/04/10 09:05:33 UTC

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats 

liuyao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17299


Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................

IMPALA-10652: Optimize the checking of the size of incremental stats

Modify the estimation method of incremental statistics size:
incremental statistics size = Existing partition statistics
    + This time calculation partition stats
    - Repeated calculation partition stats

Testing:
All partitions of a table have no incremental stats.
--Calculate the incremental stats of all partitions,
  the incremental stats size exceeds the threshold,
  an error is reported.
--Calculate the incremental stats of one partition,
  no error is reported.

Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
---
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
2 files changed, 45 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/17299/1
-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 4:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java:

http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@461
PS4, Line 461:             // The partition has incremental stats,
             :             // and the partition is not calculated this time
             :             if (part.hasIncrementalStats() && !partIds.contains(part.getId())) {
             :               ++numOfAllIncStatsPartitions;
             :             }
In the commit message you have the following formula:

 incremental statistics size = Existing partition statistics
    + This time calculation partition stats
    - Repeated calculation partition stats

You could also copy the formula here to make the code more verbose.

If I understand correctly, this part is the "Existing partition statistics". Please add a comment about it. It'd be nice if the code and comments would cleanly express the new formula.


http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@467
PS4, Line 467: numOfAllIncStatsPartitions += partitionSet_.getPartitions().size();
Please add comment that this is the "This time calculation partition stats" from the formula.


http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@471
PS4, Line 471: stas
nit: stats


http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:

http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1787
PS4, Line 1787:     checkComputeStatsStmt(
              :         "compute incremental stats functional.alltypes partition(year=2010, month=10)");
              :     BackendConfig.INSTANCE.getBackendCfg().setInc_stats_size_limit_bytes(bytes);
Please add a few more tests where you invoke compute incremental stats for a set of partitions.



-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Tue, 27 Apr 2021 17:47:44 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 7: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 29 Apr 2021 12:02:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17299/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:

http://gerrit.cloudera.org:8080/#/c/17299/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1798
PS5, Line 1798:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/17299/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1803
PS5, Line 1803:     
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 5
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 28 Apr 2021 06:21:38 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................

IMPALA-10652: Optimize the checking of the size of incremental stats

Modify the estimation method of incremental statistics size:
incremental statistics size = Existing partition statistics
    + This time calculation partition stats
    - Repeated calculation partition stats

Testing:
All partitions of a table have no incremental stats.
--Calculate the incremental stats of all partitions,
  the incremental stats size exceeds the threshold,
  an error is reported.
--Calculate the incremental stats of one partition,
  no error is reported.

Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Reviewed-on: http://gerrit.cloudera.org:8080/17299
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Zoltan Borok-Nagy <bo...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
2 files changed, 66 insertions(+), 1 deletion(-)

Approvals:
  Impala Public Jenkins: Verified
  Zoltan Borok-Nagy: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 8
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 7:

compare Patch Set 4 and Patch Set 7


-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 29 Apr 2021 04:36:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 3:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17299/2/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java:

http://gerrit.cloudera.org:8080/#/c/17299/2/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@448
PS2, Line 448: numOfAllIncStats
> May rename the variable to numOfAllIncStatsPartitions
Done


http://gerrit.cloudera.org:8080/#/c/17299/2/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@452
PS2, Line 452: if (partitionSet_ == null) {
             :           numOfAllIncStatsPartitions = allPartitio
> We may not need to verify the size limit when the partition set for increme
It is a pre-check on the size of incremental stats to prevent the incremental stats from occupying too much memory after calculation. If no partition is specified, all partitions are calculated. Need to check whether the incremental stats of all partitions exceeds the threshold after calculation.



-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 3
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 14 Apr 2021 06:18:18 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Tim Armstrong, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17299

to look at the new patch set (#2).

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................

IMPALA-10652: Optimize the checking of the size of incremental stats

Modify the estimation method of incremental statistics size:
incremental statistics size = Existing partition statistics
    + This time calculation partition stats
    - Repeated calculation partition stats

Testing:
All partitions of a table have no incremental stats.
--Calculate the incremental stats of all partitions,
  the incremental stats size exceeds the threshold,
  an error is reported.
--Calculate the incremental stats of one partition,
  no error is reported.

Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
---
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
2 files changed, 45 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/17299/2
-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 2
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8654/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 5
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 28 Apr 2021 06:39:20 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Qifan Chen, Zoltan Borok-Nagy, Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17299

to look at the new patch set (#6).

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................

IMPALA-10652: Optimize the checking of the size of incremental stats

Modify the estimation method of incremental statistics size:
incremental statistics size = Existing partition statistics
    + This time calculation partition stats
    - Repeated calculation partition stats

Testing:
All partitions of a table have no incremental stats.
--Calculate the incremental stats of all partitions,
  the incremental stats size exceeds the threshold,
  an error is reported.
--Calculate the incremental stats of one partition,
  no error is reported.

Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
---
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
2 files changed, 65 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/17299/6
-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Qifan Chen, Zoltan Borok-Nagy, Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17299

to look at the new patch set (#7).

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................

IMPALA-10652: Optimize the checking of the size of incremental stats

Modify the estimation method of incremental statistics size:
incremental statistics size = Existing partition statistics
    + This time calculation partition stats
    - Repeated calculation partition stats

Testing:
All partitions of a table have no incremental stats.
--Calculate the incremental stats of all partitions,
  the incremental stats size exceeds the threshold,
  an error is reported.
--Calculate the incremental stats of one partition,
  no error is reported.

Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
---
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
2 files changed, 66 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/17299/7
-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7103/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Tue, 27 Apr 2021 04:07:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17299/2/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java:

http://gerrit.cloudera.org:8080/#/c/17299/2/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@448
PS2, Line 448: incPartitionSize
May rename the variable to numOfAllIncStatsPartitions


http://gerrit.cloudera.org:8080/#/c/17299/2/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@452
PS2, Line 452: if (partitionSet_ == null) {
             :           incPartitionSize = allPartitions.size();
We may not need to verify the size limit when the partition set for incremental stats update is empty. The reason is that any incremental stats update done before has already passed the size test.



-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 2
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Apr 2021 16:49:57 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Qifan Chen, Tim Armstrong, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17299

to look at the new patch set (#3).

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................

IMPALA-10652: Optimize the checking of the size of incremental stats

Modify the estimation method of incremental statistics size:
incremental statistics size = Existing partition statistics
    + This time calculation partition stats
    - Repeated calculation partition stats

Testing:
All partitions of a table have no incremental stats.
--Calculate the incremental stats of all partitions,
  the incremental stats size exceeds the threshold,
  an error is reported.
--Calculate the incremental stats of one partition,
  no error is reported.

Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
---
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
2 files changed, 45 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/17299/3
-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 3
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8539/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 10 Apr 2021 09:25:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7112/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 29 Apr 2021 06:08:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8586/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 15 Apr 2021 02:46:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 6:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java:

http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@461
PS4, Line 461:           // incremental statistics size = Existing partition statistics
             :           //     - Repeated calculation partition stats
             :           //     + This time calculation partition stats
             :           for (FeFsPartition part: allPartitions) {
             :             /
> In the commit message you have the following formula:
Done


http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@467
PS4, Line 467:   // - Repeated calculation partition stats"
> Please add comment that this is the "This time calculation partition stats"
Done


http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@471
PS4, Line 471: 
> nit: stats
Done


http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:

http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1787
PS4, Line 1787:     checkComputeStatsStmt(
              :         "compute incremental stats functional.alltypes partition(year=2010, month=10)");
              : 
> Please add a few more tests where you invoke compute incremental stats for 
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 28 Apr 2021 06:39:48 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Qifan Chen, Zoltan Borok-Nagy, Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17299

to look at the new patch set (#5).

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................

IMPALA-10652: Optimize the checking of the size of incremental stats

Modify the estimation method of incremental statistics size:
incremental statistics size = Existing partition statistics
    + This time calculation partition stats
    - Repeated calculation partition stats

Testing:
All partitions of a table have no incremental stats.
--Calculate the incremental stats of all partitions,
  the incremental stats size exceeds the threshold,
  an error is reported.
--Calculate the incremental stats of one partition,
  no error is reported.

Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
---
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
2 files changed, 66 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/17299/5
-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 5
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Qifan Chen, Tim Armstrong, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17299

to look at the new patch set (#4).

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................

IMPALA-10652: Optimize the checking of the size of incremental stats

Modify the estimation method of incremental statistics size:
incremental statistics size = Existing partition statistics
    + This time calculation partition stats
    - Repeated calculation partition stats

Testing:
All partitions of a table have no incremental stats.
--Calculate the incremental stats of all partitions,
  the incremental stats size exceeds the threshold,
  an error is reported.
--Calculate the incremental stats of one partition,
  no error is reported.

Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
---
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
2 files changed, 45 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/17299/4
-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8656/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 28 Apr 2021 07:36:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17299/3/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:

http://gerrit.cloudera.org:8080/#/c/17299/3/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1784
PS3, Line 1784: fu
> nit. table alltypes?
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 15 Apr 2021 02:27:43 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8655/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 28 Apr 2021 06:43:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8540/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 2
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 10 Apr 2021 09:32:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 3: Code-Review+1

(2 comments)

Looks good!

http://gerrit.cloudera.org:8080/#/c/17299/2/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java:

http://gerrit.cloudera.org:8080/#/c/17299/2/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@452
PS2, Line 452: if (partitionSet_ == null) {
             :           numOfAllIncStatsPartitions = allPartitio
> It is a pre-check on the size of incremental stats to prevent the increment
Okay. 

Done.


http://gerrit.cloudera.org:8080/#/c/17299/3/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:

http://gerrit.cloudera.org:8080/#/c/17299/3/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1784
PS3, Line 1784: aa
nit. table alltypes?



-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 3
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 14 Apr 2021 13:04:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 7:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java:

http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@461
PS4, Line 461:           // incremental statistics size = Existing partition statistics
             :           //     - Repeated calculation partition stats
             :           //     + This time calculation partition stats
             :           for (FeFsPartition part: allPartitions) {
             :             /
> In the commit message you have the following formula:
Done


http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@467
PS4, Line 467:   // - Repeated calculation partition stats"
> Please add comment that this is the "This time calculation partition stats"
Done


http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@471
PS4, Line 471: 
> nit: stats
Done


http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:

http://gerrit.cloudera.org:8080/#/c/17299/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1787
PS4, Line 1787:     checkComputeStatsStmt(
              :         "compute incremental stats functional.alltypes partition(year=2010, month=10)");
              : 
> Please add a few more tests where you invoke compute incremental stats for 
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 29 Apr 2021 02:23:53 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 1:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/17299/1/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java:

http://gerrit.cloudera.org:8080/#/c/17299/1/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@451
PS1, Line 451:         
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/17299/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:

http://gerrit.cloudera.org:8080/#/c/17299/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1783
PS1, Line 1783:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/17299/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1785
PS1, Line 1785:     // so if only one partition is calculated, 
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 10 Apr 2021 09:06:16 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 7: Code-Review+2

Thanks for the change. LGTM!


-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 29 Apr 2021 13:13:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10652: Optimize the checking of the size of incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17299 )

Change subject: IMPALA-10652: Optimize the checking of the size of incremental stats
......................................................................


Patch Set 4: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f35ea936445015a3b8b8102b1891db29751b5ee
Gerrit-Change-Number: 17299
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Tue, 27 Apr 2021 09:51:55 +0000
Gerrit-HasComments: No