You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org> on 2018/09/12 19:58:45 UTC
[Impala-ASF-CR] IMPALA-3819: Clarify stale block metadata warning message
Bharath Vissapragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11433
Change subject: IMPALA-3819: Clarify stale block metadata warning message
......................................................................
IMPALA-3819: Clarify stale block metadata warning message
We got user reports that the warning we display for stale block
metadata is confusing and gives an impression that it can cause
incorrect results. This commit clarifies the warning message.
Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
---
M be/src/exec/hdfs-scan-node-base.cc
1 file changed, 5 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/11433/1
--
To view, visit http://gerrit.cloudera.org:8080/11433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Gerrit-Change-Number: 11433
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
[Impala-ASF-CR] IMPALA-3819: Clarify stale block metadata warning message
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11433 )
Change subject: IMPALA-3819: Clarify stale block metadata warning message
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/11433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Gerrit-Change-Number: 11433
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Sep 2018 20:30:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3819: Clarify stale block metadata warning message
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11433 )
Change subject: IMPALA-3819: Clarify stale block metadata warning message
......................................................................
IMPALA-3819: Clarify stale block metadata warning message
We got user reports that the warning we display for stale block
metadata is confusing and gives an impression that it can cause
incorrect results. This commit clarifies the warning message.
Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Reviewed-on: http://gerrit.cloudera.org:8080/11433
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/hdfs-scan-node-base.cc
1 file changed, 5 insertions(+), 3 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/11433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Gerrit-Change-Number: 11433
Gerrit-PatchSet: 4
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
[Impala-ASF-CR] IMPALA-3819: Clarify stale block metadata warning message
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11433 )
Change subject: IMPALA-3819: Clarify stale block metadata warning message
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3154/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/11433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Gerrit-Change-Number: 11433
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Sep 2018 16:58:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3819: Clarify stale block metadata warning message
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11433 )
Change subject: IMPALA-3819: Clarify stale block metadata warning message
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/663/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Gerrit-Change-Number: 11433
Gerrit-PatchSet: 2
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Sep 2018 17:28:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3819: Clarify stale block metadata warning message
Posted by "Zoram Thanga (Code Review)" <ge...@cloudera.org>.
Zoram Thanga has posted comments on this change. ( http://gerrit.cloudera.org:8080/11433 )
Change subject: IMPALA-3819: Clarify stale block metadata warning message
......................................................................
Patch Set 1: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11433/1/be/src/exec/hdfs-scan-node-base.cc
File be/src/exec/hdfs-scan-node-base.cc:
http://gerrit.cloudera.org:8080/#/c/11433/1/be/src/exec/hdfs-scan-node-base.cc@913
PS1, Line 913: runtime_state_->LogError(ErrorMsg(TErrorCode::GENERAL, Substitute(
> These warnings show up in the runtime profile (warnings section) and the sh
Ok. Your explanation makes sense. I rest my case :)
http://gerrit.cloudera.org:8080/#/c/11433/1/be/src/exec/hdfs-scan-node-base.cc@914
PS1, Line 914: "Read $0 of data across network that was expected to be local. Block locality "
> Same as above.
Ok.
--
To view, visit http://gerrit.cloudera.org:8080/11433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Gerrit-Change-Number: 11433
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Sep 2018 20:39:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3819: Clarify stale block metadata warning message
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Hello Zoram Thanga, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11433
to look at the new patch set (#2).
Change subject: IMPALA-3819: Clarify stale block metadata warning message
......................................................................
IMPALA-3819: Clarify stale block metadata warning message
We got user reports that the warning we display for stale block
metadata is confusing and gives an impression that it can cause
incorrect results. This commit clarifies the warning message.
Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
---
M be/src/exec/hdfs-scan-node-base.cc
1 file changed, 5 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/11433/2
--
To view, visit http://gerrit.cloudera.org:8080/11433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Gerrit-Change-Number: 11433
Gerrit-PatchSet: 2
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
[Impala-ASF-CR] IMPALA-3819: Clarify stale block metadata warning message
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11433 )
Change subject: IMPALA-3819: Clarify stale block metadata warning message
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/653/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Gerrit-Change-Number: 11433
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Sep 2018 20:34:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3819: Clarify stale block metadata warning message
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11433 )
Change subject: IMPALA-3819: Clarify stale block metadata warning message
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11433/1/be/src/exec/hdfs-scan-node-base.cc
File be/src/exec/hdfs-scan-node-base.cc:
http://gerrit.cloudera.org:8080/#/c/11433/1/be/src/exec/hdfs-scan-node-base.cc@917
PS1, Line 917: shuffling the underlying blocks around the cluste
> possible alt: "moving some of the file's blocks."
Done
--
To view, visit http://gerrit.cloudera.org:8080/11433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Gerrit-Change-Number: 11433
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Sep 2018 16:57:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3819: Clarify stale block metadata warning message
Posted by "Zoram Thanga (Code Review)" <ge...@cloudera.org>.
Zoram Thanga has posted comments on this change. ( http://gerrit.cloudera.org:8080/11433 )
Change subject: IMPALA-3819: Clarify stale block metadata warning message
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11433/1/be/src/exec/hdfs-scan-node-base.cc
File be/src/exec/hdfs-scan-node-base.cc:
http://gerrit.cloudera.org:8080/#/c/11433/1/be/src/exec/hdfs-scan-node-base.cc@913
PS1, Line 913: runtime_state_->LogError(ErrorMsg(TErrorCode::GENERAL, Substitute(
If this does not result in incorrect query results, I wonder if logging it as an error is appropriate. IMO this is probably a WARN or INFO message.
http://gerrit.cloudera.org:8080/#/c/11433/1/be/src/exec/hdfs-scan-node-base.cc@914
PS1, Line 914: "Read $0 of data across network that was expected to be local. Block locality "
Worth prefixing the message with the query/Finst Id?
--
To view, visit http://gerrit.cloudera.org:8080/11433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Gerrit-Change-Number: 11433
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Sep 2018 20:17:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3819: Clarify stale block metadata warning message
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11433 )
Change subject: IMPALA-3819: Clarify stale block metadata warning message
......................................................................
Patch Set 1: Code-Review+2
(1 comment)
Thanks for fixing, always good to avoid confusion where possible!
http://gerrit.cloudera.org:8080/#/c/11433/1/be/src/exec/hdfs-scan-node-base.cc
File be/src/exec/hdfs-scan-node-base.cc:
http://gerrit.cloudera.org:8080/#/c/11433/1/be/src/exec/hdfs-scan-node-base.cc@917
PS1, Line 917: shuffling the underlying blocks around the cluste
possible alt: "moving some of the file's blocks."
My thought process is that plainer language might be clearer. But I'm ok with the current message, it's a clear improvement.
--
To view, visit http://gerrit.cloudera.org:8080/11433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Gerrit-Change-Number: 11433
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Sep 2018 15:08:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3819: Clarify stale block metadata warning message
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11433 )
Change subject: IMPALA-3819: Clarify stale block metadata warning message
......................................................................
Patch Set 2: Code-Review+2
Carrying +2.
--
To view, visit http://gerrit.cloudera.org:8080/11433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Gerrit-Change-Number: 11433
Gerrit-PatchSet: 2
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Sep 2018 16:58:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3819: Clarify stale block metadata warning message
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11433 )
Change subject: IMPALA-3819: Clarify stale block metadata warning message
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11433/1/be/src/exec/hdfs-scan-node-base.cc
File be/src/exec/hdfs-scan-node-base.cc:
http://gerrit.cloudera.org:8080/#/c/11433/1/be/src/exec/hdfs-scan-node-base.cc@913
PS1, Line 913: runtime_state_->LogError(ErrorMsg(TErrorCode::GENERAL, Substitute(
> If this does not result in incorrect query results, I wonder if logging it
These warnings show up in the runtime profile (warnings section) and the shell, which I think are the right places to bubble up these alerts.
Reason being, if the end user suspects that the query has suddenly become slower, they might see these warnings in the profile and figure out there are remote reads. OTOH flushing it to the log file makes it difficult to map the entries to slow query IDs (extra hops). Thoughts?
http://gerrit.cloudera.org:8080/#/c/11433/1/be/src/exec/hdfs-scan-node-base.cc@914
PS1, Line 914: "Read $0 of data across network that was expected to be local. Block locality "
> Worth prefixing the message with the query/Finst Id?
Same as above.
--
To view, visit http://gerrit.cloudera.org:8080/11433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Gerrit-Change-Number: 11433
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Sep 2018 20:35:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3819: Clarify stale block metadata warning message
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11433 )
Change subject: IMPALA-3819: Clarify stale block metadata warning message
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/11433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9eb2672393c5ff4e0670d247d684278283012a3
Gerrit-Change-Number: 11433
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoram Thanga <zo...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Sep 2018 16:58:17 +0000
Gerrit-HasComments: No