You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org> on 2018/09/19 19:28:55 UTC

[Impala-ASF-CR] IMPALA-7498: Fix log spew from LocalCatalog startup 

Bharath Vissapragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11472


Change subject: IMPALA-7498: Fix log spew from LocalCatalog startup
......................................................................

IMPALA-7498: Fix log spew from LocalCatalog startup

Frontend calls LocalCatalog#waitForCatalog() in a tight loop during
startup and ends up spewing tons of log messages if the MetaProvider
takes some time to initialize.

This is a problem for CatalogdMetaProvider implementation since it
waits on the Catalog server to send an initial update. This can take
some time depending on various factors and the logs become too noisy
during that period.

This patch adds a sleep() inside waitForCatalog() to avoid this. We
can do something fancy, like ImpaladCatalog implementation, by waiting
on a synchronized monitor and getting notified when the Catalog is
initialized, but I don't think it is worth the effort, especially since
the sleep time is pretty small (MAX_CATALOG_UPDATE_WAIT_TIME_MS = 2s).

Testing: Tested it locally by just starting the impalad catalog in local
mode and the logs look much better while waiting for the Catalog to send
the initial update.

Change-Id: I81e7cc6f464fd71afecd94a5a2b777aff47ee11b
---
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
1 file changed, 6 insertions(+), 4 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/11472/1
-- 
To view, visit http://gerrit.cloudera.org:8080/11472
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I81e7cc6f464fd71afecd94a5a2b777aff47ee11b
Gerrit-Change-Number: 11472
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>

[Impala-ASF-CR] IMPALA-7498: Fix log spew from LocalCatalog startup

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11472 )

Change subject: IMPALA-7498: Fix log spew from LocalCatalog startup
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3198/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/11472
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81e7cc6f464fd71afecd94a5a2b777aff47ee11b
Gerrit-Change-Number: 11472
Gerrit-PatchSet: 2
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Sep 2018 06:44:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7498: Fix log spew from LocalCatalog startup

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11472 )

Change subject: IMPALA-7498: Fix log spew from LocalCatalog startup
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/11472
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81e7cc6f464fd71afecd94a5a2b777aff47ee11b
Gerrit-Change-Number: 11472
Gerrit-PatchSet: 2
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Sep 2018 10:28:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7498: Fix log spew from LocalCatalog startup

Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, Vuk Ercegovac, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/11472

to look at the new patch set (#2).

Change subject: IMPALA-7498: Fix log spew from LocalCatalog startup
......................................................................

IMPALA-7498: Fix log spew from LocalCatalog startup

Frontend calls LocalCatalog#waitForCatalog() in a tight loop during
startup and ends up spewing tons of log messages if the MetaProvider
takes some time to initialize.

This is a problem for CatalogdMetaProvider implementation since it
waits on the Catalog server to send an initial update. This can take
some time depending on various factors and the logs become too noisy
during that period.

This patch adds a sleep() inside waitForCatalog() to avoid this. We
can do something fancy, like ImpaladCatalog implementation, by waiting
on a synchronized monitor and getting notified when the Catalog is
initialized, but I don't think it is worth the effort, especially since
the sleep time is pretty small (MAX_CATALOG_UPDATE_WAIT_TIME_MS = 2s).

Testing: Tested it locally by just starting the impalad catalog in local
mode and the logs look much better while waiting for the Catalog to send
the initial update.

Change-Id: I81e7cc6f464fd71afecd94a5a2b777aff47ee11b
---
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
1 file changed, 7 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/11472/2
-- 
To view, visit http://gerrit.cloudera.org:8080/11472
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I81e7cc6f464fd71afecd94a5a2b777aff47ee11b
Gerrit-Change-Number: 11472
Gerrit-PatchSet: 2
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>

[Impala-ASF-CR] IMPALA-7498: Fix log spew from LocalCatalog startup

Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11472 )

Change subject: IMPALA-7498: Fix log spew from LocalCatalog startup
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/11472/1/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java:

http://gerrit.cloudera.org:8080/#/c/11472/1/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java@213
PS1, Line 213:       Thread.sleep(timeoutMs);
looked at all call-sites... is the one in StmtLoader going to do an unneeded sleep here? I think that sleep is set to 2s. perhaps get out of here early with a check via isReady?



-- 
To view, visit http://gerrit.cloudera.org:8080/11472
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81e7cc6f464fd71afecd94a5a2b777aff47ee11b
Gerrit-Change-Number: 11472
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Fri, 21 Sep 2018 16:29:16 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7498: Fix log spew from LocalCatalog startup

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11472 )

Change subject: IMPALA-7498: Fix log spew from LocalCatalog startup
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/715/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11472
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81e7cc6f464fd71afecd94a5a2b777aff47ee11b
Gerrit-Change-Number: 11472
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Sep 2018 20:01:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7498: Fix log spew from LocalCatalog startup

Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11472 )

Change subject: IMPALA-7498: Fix log spew from LocalCatalog startup
......................................................................

IMPALA-7498: Fix log spew from LocalCatalog startup

Frontend calls LocalCatalog#waitForCatalog() in a tight loop during
startup and ends up spewing tons of log messages if the MetaProvider
takes some time to initialize.

This is a problem for CatalogdMetaProvider implementation since it
waits on the Catalog server to send an initial update. This can take
some time depending on various factors and the logs become too noisy
during that period.

This patch adds a sleep() inside waitForCatalog() to avoid this. We
can do something fancy, like ImpaladCatalog implementation, by waiting
on a synchronized monitor and getting notified when the Catalog is
initialized, but I don't think it is worth the effort, especially since
the sleep time is pretty small (MAX_CATALOG_UPDATE_WAIT_TIME_MS = 2s).

Testing: Tested it locally by just starting the impalad catalog in local
mode and the logs look much better while waiting for the Catalog to send
the initial update.

Change-Id: I81e7cc6f464fd71afecd94a5a2b777aff47ee11b
Reviewed-on: http://gerrit.cloudera.org:8080/11472
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Vuk Ercegovac <ve...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
1 file changed, 7 insertions(+), 4 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Vuk Ercegovac: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/11472
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I81e7cc6f464fd71afecd94a5a2b777aff47ee11b
Gerrit-Change-Number: 11472
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>

[Impala-ASF-CR] IMPALA-7498: Fix log spew from LocalCatalog startup

Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11472 )

Change subject: IMPALA-7498: Fix log spew from LocalCatalog startup
......................................................................


Patch Set 2: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/11472
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81e7cc6f464fd71afecd94a5a2b777aff47ee11b
Gerrit-Change-Number: 11472
Gerrit-PatchSet: 2
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Sep 2018 15:59:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7498: Fix log spew from LocalCatalog startup

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11472 )

Change subject: IMPALA-7498: Fix log spew from LocalCatalog startup
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/755/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11472
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81e7cc6f464fd71afecd94a5a2b777aff47ee11b
Gerrit-Change-Number: 11472
Gerrit-PatchSet: 2
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Sep 2018 06:39:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7498: Fix log spew from LocalCatalog startup

Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11472 )

Change subject: IMPALA-7498: Fix log spew from LocalCatalog startup
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/11472/1/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java:

http://gerrit.cloudera.org:8080/#/c/11472/1/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java@213
PS1, Line 213:       Thread.sleep(timeoutMs);
> looked at all call-sites... is the one in StmtLoader going to do an unneede
Missed this caller, done.



-- 
To view, visit http://gerrit.cloudera.org:8080/11472
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81e7cc6f464fd71afecd94a5a2b777aff47ee11b
Gerrit-Change-Number: 11472
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Sep 2018 06:03:17 +0000
Gerrit-HasComments: Yes