You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Alex Behm (Code Review)" <ge...@cloudera.org> on 2018/03/15 17:54:29 UTC

[Impala-ASF-CR] IMPALA-6675: Default to --compact catalog topic=true.

Alex Behm has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9661


Change subject: IMPALA-6675: Default to --compact_catalog_topic=true.
......................................................................

IMPALA-6675: Default to --compact_catalog_topic=true.

Testing:
- Ran a few queries locally
- Ran test_compact_catalog_updates.py locally
- Waiting for a perf/safety evaluation from Mostafa

Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
---
M be/src/common/global-flags.cc
M tests/custom_cluster/test_compact_catalog_updates.py
2 files changed, 4 insertions(+), 4 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/9661/1
-- 
To view, visit http://gerrit.cloudera.org:8080/9661
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
Gerrit-Change-Number: 9661
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Behm <al...@cloudera.com>

[Impala-ASF-CR] IMPALA-6675: Default to --compact catalog topic=true.

Posted by "Mostafa Mokhtar (Code Review)" <ge...@cloudera.org>.
Mostafa Mokhtar has posted comments on this change. ( http://gerrit.cloudera.org:8080/9661 )

Change subject: IMPALA-6675: Default to --compact_catalog_topic=true.
......................................................................


Patch Set 2:

Ran small and large payload test cases against a 130 node cluster, speedup from compaction and reduction in network traffic are good.


-- 
To view, visit http://gerrit.cloudera.org:8080/9661
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
Gerrit-Change-Number: 9661
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mm...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Mar 2018 15:28:10 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6675: Default to --compact catalog topic=true.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9661 )

Change subject: IMPALA-6675: Default to --compact_catalog_topic=true.
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/9661
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
Gerrit-Change-Number: 9661
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Mostafa Mokhtar <mm...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Mar 2018 19:12:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6675: Default to --compact catalog topic=true.

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/9661 )

Change subject: IMPALA-6675: Default to --compact_catalog_topic=true.
......................................................................


Patch Set 2: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/9661
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
Gerrit-Change-Number: 9661
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 Mar 2018 20:18:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6675: Default to --compact catalog topic=true.

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/9661 )

Change subject: IMPALA-6675: Default to --compact_catalog_topic=true.
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/9661
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
Gerrit-Change-Number: 9661
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mm...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Mar 2018 15:32:26 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6675: Default to --compact catalog topic=true.

Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/9661 )

Change subject: IMPALA-6675: Default to --compact_catalog_topic=true.
......................................................................


Patch Set 1: Code-Review+1

(2 comments)

http://gerrit.cloudera.org:8080/#/c/9661/1/tests/custom_cluster/test_compact_catalog_updates.py
File tests/custom_cluster/test_compact_catalog_updates.py:

http://gerrit.cloudera.org:8080/#/c/9661/1/tests/custom_cluster/test_compact_catalog_updates.py@18
PS1, Line 18: true
false


http://gerrit.cloudera.org:8080/#/c/9661/1/tests/custom_cluster/test_compact_catalog_updates.py@28
PS1, Line 28: test_compact_catalog_topic_updates
nit: test_non_compact_..



-- 
To view, visit http://gerrit.cloudera.org:8080/9661
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
Gerrit-Change-Number: 9661
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 Mar 2018 19:08:28 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6675: Default to --compact catalog topic=true.

Posted by "Mostafa Mokhtar (Code Review)" <ge...@cloudera.org>.
Mostafa Mokhtar has posted comments on this change. ( http://gerrit.cloudera.org:8080/9661 )

Change subject: IMPALA-6675: Default to --compact_catalog_topic=true.
......................................................................


Patch Set 2: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/9661
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
Gerrit-Change-Number: 9661
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mm...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Mar 2018 15:27:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6675: Default to --compact catalog topic=true.

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/9661

to look at the new patch set (#2).

Change subject: IMPALA-6675: Default to --compact_catalog_topic=true.
......................................................................

IMPALA-6675: Default to --compact_catalog_topic=true.

Testing:
- Ran a few queries locally
- Ran test_compact_catalog_updates.py locally
- Waiting for a perf/safety evaluation from Mostafa

Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
---
M be/src/common/global-flags.cc
M tests/custom_cluster/test_compact_catalog_updates.py
2 files changed, 6 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/9661/2
-- 
To view, visit http://gerrit.cloudera.org:8080/9661
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
Gerrit-Change-Number: 9661
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>

[Impala-ASF-CR] IMPALA-6675: Default to --compact catalog topic=true.

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/9661 )

Change subject: IMPALA-6675: Default to --compact_catalog_topic=true.
......................................................................


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/9661/1/tests/custom_cluster/test_compact_catalog_updates.py
File tests/custom_cluster/test_compact_catalog_updates.py:

http://gerrit.cloudera.org:8080/#/c/9661/1/tests/custom_cluster/test_compact_catalog_updates.py@18
PS1, Line 18: true
> false
Done


http://gerrit.cloudera.org:8080/#/c/9661/1/tests/custom_cluster/test_compact_catalog_updates.py@28
PS1, Line 28: test_compact_catalog_topic_updates
> nit: test_non_compact_..
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/9661
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
Gerrit-Change-Number: 9661
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 Mar 2018 20:13:13 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6675: Default to --compact catalog topic=true.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9661 )

Change subject: IMPALA-6675: Default to --compact_catalog_topic=true.
......................................................................

IMPALA-6675: Default to --compact_catalog_topic=true.

Testing:
- Ran a few queries locally
- Ran test_compact_catalog_updates.py locally

Mostafa's perf evaluation:
- 130 node cluster
- Load metadata after invalidate for 4 tables, each
  with 100K partitions and 1 million files

Results compaction on vs. compaction off
- 5.7x reduction in topic size and Network
- 30% reduction in Catalog+Statestore CPU
- 15% speedup in query time
- Compaction of topic takes ~22s in the Catalog
- Time spent by Statestore sending the topics is
  reduced from 90s to 17s
- Max topic update duration reduced from 72s to 11s

Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
Reviewed-on: http://gerrit.cloudera.org:8080/9661
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/common/global-flags.cc
M tests/custom_cluster/test_compact_catalog_updates.py
2 files changed, 6 insertions(+), 6 deletions(-)

Approvals:
  Alex Behm: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/9661
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
Gerrit-Change-Number: 9661
Gerrit-PatchSet: 4
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Mostafa Mokhtar <mm...@cloudera.com>

[Impala-ASF-CR] IMPALA-6675: Default to --compact catalog topic=true.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9661 )

Change subject: IMPALA-6675: Default to --compact_catalog_topic=true.
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2119/


-- 
To view, visit http://gerrit.cloudera.org:8080/9661
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
Gerrit-Change-Number: 9661
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Mostafa Mokhtar <mm...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Mar 2018 15:33:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6675: Default to --compact catalog topic=true.

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Mostafa Mokhtar, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/9661

to look at the new patch set (#3).

Change subject: IMPALA-6675: Default to --compact_catalog_topic=true.
......................................................................

IMPALA-6675: Default to --compact_catalog_topic=true.

Testing:
- Ran a few queries locally
- Ran test_compact_catalog_updates.py locally

Mostafa's perf evaluation:
- 130 node cluster
- Load metadata after invalidate for 4 tables, each
  with 100K partitions and 1 million files

Results compaction on vs. compaction off
- 5.7x reduction in topic size and Network
- 30% reduction in Catalog+Statestore CPU
- 15% speedup in query time
- Compaction of topic takes ~22s in the Catalog
- Time spent by Statestore sending the topics is
  reduced from 90s to 17s
- Max topic update duration reduced from 72s to 11s

Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
---
M be/src/common/global-flags.cc
M tests/custom_cluster/test_compact_catalog_updates.py
2 files changed, 6 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/9661/3
-- 
To view, visit http://gerrit.cloudera.org:8080/9661
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I39a2dd42a21ef448b85278a8cef3c1d0112b844f
Gerrit-Change-Number: 9661
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mm...@cloudera.com>