You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Kurt Deschler (Code Review)" <ge...@cloudera.org> on 2021/03/17 19:41:34 UTC
[Impala-ASF-CR] IMPALA-10579: Avoid infinite loop from remote file iterator
Kurt Deschler has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17192
Change subject: IMPALA-10579: Avoid infinite loop from remote file iterator
......................................................................
IMPALA-10579: Avoid infinite loop from remote file iterator
This patch adds a flag to FileSystemUtil.FilterIterator.hasNext() to
avoid getting into an infinite loop if the remote iterator throws
repeated FileNotFoundException exceptions. Since these exceptions do not
contain information regarding which file is missing, the new check simply
re-throws the exception if the remote hasNext() call throws twice in a
row.
Testing:
Tested manually by temporarily adding explicit exceptions in the
problem loop
Change-Id: Ia8b60aab0286390ced364188a24ee78f99073730
---
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
1 file changed, 10 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/17192/1
--
To view, visit http://gerrit.cloudera.org:8080/17192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia8b60aab0286390ced364188a24ee78f99073730
Gerrit-Change-Number: 17192
Gerrit-PatchSet: 1
Gerrit-Owner: Kurt Deschler <kd...@cloudera.com>
[Impala-ASF-CR] IMPALA-10579: Avoid infinite loop from remote file iterator
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17192 )
Change subject: IMPALA-10579: Avoid infinite loop from remote file iterator
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/17192/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java:
http://gerrit.cloudera.org:8080/#/c/17192/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@837
PS1, Line 837: IMPALA-1057
IMPALA-10579
http://gerrit.cloudera.org:8080/#/c/17192/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@840
PS1, Line 840: if (last_iter_had_exception) {
: throw ex;
: }
I think if the iterator encounters two absence subdirectories continuely, we will throw exception here. E.g. when there are three concurrent INSERTs on the table, the last finished INSERT could fail by this.
Could you loop test_insert_stress.py several times to test?
--
To view, visit http://gerrit.cloudera.org:8080/17192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8b60aab0286390ced364188a24ee78f99073730
Gerrit-Change-Number: 17192
Gerrit-PatchSet: 1
Gerrit-Owner: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 18 Mar 2021 00:38:49 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10579: Avoid infinite loop from remote file iterator
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17192 )
Change subject: IMPALA-10579: Avoid infinite loop from remote file iterator
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8384/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8b60aab0286390ced364188a24ee78f99073730
Gerrit-Change-Number: 17192
Gerrit-PatchSet: 1
Gerrit-Owner: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Mar 2021 20:02:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10579: Avoid infinite loop from remote file iterator
Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.
Kurt Deschler has abandoned this change. ( http://gerrit.cloudera.org:8080/17192 )
Change subject: IMPALA-10579: Avoid infinite loop from remote file iterator
......................................................................
Abandoned
--
To view, visit http://gerrit.cloudera.org:8080/17192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: Ia8b60aab0286390ced364188a24ee78f99073730
Gerrit-Change-Number: 17192
Gerrit-PatchSet: 1
Gerrit-Owner: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>