You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org> on 2018/10/12 18:15:19 UTC

[Impala-ASF-CR] IMPALA-7622: adds profile metrics for incremental stats 

Vuk Ercegovac has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11670


Change subject: IMPALA-7622: adds profile metrics for incremental stats
......................................................................

IMPALA-7622: adds profile metrics for incremental stats

Reapplies change after fixing where frontend profile is placed in runtime
profile.

When computing incremental statistics by fetching the stats directly
from catalogd, a potentially expensive RPC is made from the impalad
coordinator to catalogd. This change adds metrics to the frontend
section of the profile to track how long the request takes, the size
of the compressed bytes received, and the number of partitions received.

The profile for a 'compute incremental ...' command on a table with
no statistics looks like this:

Frontend:
     - StatsFetch.CompressedBytes: 0
     - StatsFetch.TotalPartitions: 24
     - StatsFetch.NumPartitionsWithStats: 0
     - StatsFetch.Time: 26ms

And the profile looks as follows when the table has stats, so the stats
are fetched:

Frontend:
     - StatsFetch.CompressedBytes: 24622
     - StatsFetch.TotalPartitions: 23
     - StatsFetch.NumPartitionsWithStats: 23
     - StatsFetch.Time: 14ms

Testing:
- manual inspection
- e2e test to check the profile

Change-Id: I94559a749500d44aa6aad564134d55c39e1d5273
---
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M tests/common/custom_cluster_test_suite.py
M tests/custom_cluster/test_pull_stats.py
3 files changed, 93 insertions(+), 3 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/11670/1
-- 
To view, visit http://gerrit.cloudera.org:8080/11670
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I94559a749500d44aa6aad564134d55c39e1d5273
Gerrit-Change-Number: 11670
Gerrit-PatchSet: 1
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>

[Impala-ASF-CR] IMPALA-7622: adds profile metrics for incremental stats

Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11670 )

Change subject: IMPALA-7622: adds profile metrics for incremental stats
......................................................................


Patch Set 1:

correct.


-- 
To view, visit http://gerrit.cloudera.org:8080/11670
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I94559a749500d44aa6aad564134d55c39e1d5273
Gerrit-Change-Number: 11670
Gerrit-PatchSet: 1
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Oct 2018 19:54:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7622: adds profile metrics for incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11670 )

Change subject: IMPALA-7622: adds profile metrics for incremental stats
......................................................................


Patch Set 1: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/11670
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I94559a749500d44aa6aad564134d55c39e1d5273
Gerrit-Change-Number: 11670
Gerrit-PatchSet: 1
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Oct 2018 23:44:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7622: adds profile metrics for incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11670 )

Change subject: IMPALA-7622: adds profile metrics for incremental stats
......................................................................

IMPALA-7622: adds profile metrics for incremental stats

Reapplies change after fixing where frontend profile is placed in runtime
profile.

When computing incremental statistics by fetching the stats directly
from catalogd, a potentially expensive RPC is made from the impalad
coordinator to catalogd. This change adds metrics to the frontend
section of the profile to track how long the request takes, the size
of the compressed bytes received, and the number of partitions received.

The profile for a 'compute incremental ...' command on a table with
no statistics looks like this:

Frontend:
     - StatsFetch.CompressedBytes: 0
     - StatsFetch.TotalPartitions: 24
     - StatsFetch.NumPartitionsWithStats: 0
     - StatsFetch.Time: 26ms

And the profile looks as follows when the table has stats, so the stats
are fetched:

Frontend:
     - StatsFetch.CompressedBytes: 24622
     - StatsFetch.TotalPartitions: 23
     - StatsFetch.NumPartitionsWithStats: 23
     - StatsFetch.Time: 14ms

Testing:
- manual inspection
- e2e test to check the profile

Change-Id: I94559a749500d44aa6aad564134d55c39e1d5273
Reviewed-on: http://gerrit.cloudera.org:8080/11670
Reviewed-by: Tianyi Wang <tw...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M tests/common/custom_cluster_test_suite.py
M tests/custom_cluster/test_pull_stats.py
3 files changed, 93 insertions(+), 3 deletions(-)

Approvals:
  Tianyi Wang: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/11670
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I94559a749500d44aa6aad564134d55c39e1d5273
Gerrit-Change-Number: 11670
Gerrit-PatchSet: 2
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>

[Impala-ASF-CR] IMPALA-7622: adds profile metrics for incremental stats

Posted by "Tianyi Wang (Code Review)" <ge...@cloudera.org>.
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11670 )

Change subject: IMPALA-7622: adds profile metrics for incremental stats
......................................................................


Patch Set 1: Code-Review+2

This was reverted only to revert IMPALA-7527, right?


-- 
To view, visit http://gerrit.cloudera.org:8080/11670
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I94559a749500d44aa6aad564134d55c39e1d5273
Gerrit-Change-Number: 11670
Gerrit-PatchSet: 1
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Oct 2018 19:07:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7622: adds profile metrics for incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11670 )

Change subject: IMPALA-7622: adds profile metrics for incremental stats
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1039/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11670
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I94559a749500d44aa6aad564134d55c39e1d5273
Gerrit-Change-Number: 11670
Gerrit-PatchSet: 1
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Oct 2018 18:43:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7622: adds profile metrics for incremental stats

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11670 )

Change subject: IMPALA-7622: adds profile metrics for incremental stats
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3307/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/11670
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I94559a749500d44aa6aad564134d55c39e1d5273
Gerrit-Change-Number: 11670
Gerrit-PatchSet: 1
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Oct 2018 19:55:57 +0000
Gerrit-HasComments: No