You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "ttttttz (Code Review)" <ge...@cloudera.org> on 2022/05/27 07:11:38 UTC
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files.
ttttttz has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18571
Change subject: IMPALA-11296: Fix infinite loop when reading orc files.
......................................................................
IMPALA-11296: Fix infinite loop when reading orc files.
Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
---
M be/src/exec/orc-column-readers.cc
1 file changed, 7 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/18571/1
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 1
Gerrit-Owner: ttttttz <24...@qq.com>
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files
......................................................................
Patch Set 8: Code-Review+2
Thanks for fixing this!
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 8
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
Gerrit-Comment-Date: Fri, 10 Jun 2022 11:22:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files
......................................................................
Patch Set 8: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 8
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
Gerrit-Comment-Date: Fri, 10 Jun 2022 15:09:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files
Posted by "ttttttz (Code Review)" <ge...@cloudera.org>.
ttttttz has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files
......................................................................
IMPALA-11296: Fix infinite loop when reading orc files
When querying an ORC table, selecting only the missing fields of ORC files causes the query to be executed indefinitely. The corresponding execution node will see some resident threads that occupy CPU abnormally. The problem is caused by this: when OrcComplexColumnReader.children_.empty() is true, OrcComplexColumnReader.row_idx_ will remain constant, causing an infinite loop at HdfsOrcScanner::TransferTuples(). We should allow empty 'children_' for original files.
Testing:
- Added a test to test_scanners.py that ensures the query can be executed successfully when selecting only the missing fields of ORC files.
Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
---
M be/src/exec/orc-column-readers.cc
M tests/query_test/test_scanners.py
2 files changed, 26 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/18571/4
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 4
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files.
Posted by "ttttttz (Code Review)" <ge...@cloudera.org>.
ttttttz has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files.
......................................................................
IMPALA-11296: Fix infinite loop when reading orc files.
Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
---
M be/src/exec/orc-column-readers.cc
M tests/query_test/test_scanners.py
2 files changed, 18 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/18571/2
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 2
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files
Posted by "ttttttz (Code Review)" <ge...@cloudera.org>.
ttttttz has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files
......................................................................
IMPALA-11296: Fix infinite loop when reading orc files
When querying an ORC table, selecting only the missing fields of ORC
files causes the query to be executed indefinitely. The
corresponding execution node will see some resident threads that
occupy CPU abnormally. The problem is caused by this:
when OrcComplexColumnReader.children_.empty() is true,
OrcComplexColumnReader.row_idx_ will remain constant, causing an
infinite loop at HdfsOrcScanner::TransferTuples().
We should allow empty 'children_' for original files.
Testing:
- Added a test to test_scanners.py that ensures the query can be
executed successfully when selecting only the missing fields of
ORC files.
Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
---
M be/src/exec/orc-column-readers.cc
M tests/query_test/test_scanners.py
2 files changed, 27 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/18571/7
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 7
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18571/5/tests/query_test/test_scanners.py
File tests/query_test/test_scanners.py:
http://gerrit.cloudera.org:8080/#/c/18571/5/tests/query_test/test_scanners.py@1438
PS5, Line 1438: n
flake8: E501 line too long (95 > 90 characters)
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 5
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
Gerrit-Comment-Date: Fri, 10 Jun 2022 03:40:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files.
Posted by "ttttttz (Code Review)" <ge...@cloudera.org>.
ttttttz has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files.
......................................................................
IMPALA-11296: Fix infinite loop when reading orc files.
Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
---
M be/src/exec/orc-column-readers.cc
M tests/query_test/test_scanners.py
2 files changed, 18 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/18571/3
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 3
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files.
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files.
......................................................................
Patch Set 3:
(7 comments)
The fix LGTM.
http://gerrit.cloudera.org:8080/#/c/18571/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18571/1//COMMIT_MSG@8
PS1, Line 8:
> Please add the commit message that is exactly long enough to explain what t
Could you please write some commit message?
http://gerrit.cloudera.org:8080/#/c/18571/3/be/src/exec/orc-column-readers.cc
File be/src/exec/orc-column-readers.cc:
http://gerrit.cloudera.org:8080/#/c/18571/3/be/src/exec/orc-column-readers.cc@410
PS3, Line 410:
nit: we use 2 spaces indention instead of 4 spaces.
http://gerrit.cloudera.org:8080/#/c/18571/3/tests/query_test/test_scanners.py
File tests/query_test/test_scanners.py:
http://gerrit.cloudera.org:8080/#/c/18571/3/tests/query_test/test_scanners.py@1423
PS3, Line 1423: field
nit: "field" -> "fields"
http://gerrit.cloudera.org:8080/#/c/18571/3/tests/query_test/test_scanners.py@1423
PS3, Line 1423: meta
nit: "meta" -> "file meta"
http://gerrit.cloudera.org:8080/#/c/18571/3/tests/query_test/test_scanners.py@1424
PS3, Line 1424: An infinite loop will occur during execution. The corresponding execution
: node will see some resident threads that occupy CPU abnormally.
Don't need these description about the bug behavior. It's fixed by this patch.
http://gerrit.cloudera.org:8080/#/c/18571/3/tests/query_test/test_scanners.py@1432
PS3, Line 1432: self.client.execute("select f1 from %s " % orc_tbl_name)
Let's also verify the results here.
result = self.client.execute("select f1 from %s " % orc_tbl_name)
assert result.data == ['NULL']
http://gerrit.cloudera.org:8080/#/c/18571/3/tests/query_test/test_scanners.py@1433
PS3, Line 1433:
Can we also test on a large table? E.g.
create external table lineitem_orc_ext like tpch_orc_def.lineitem location '/test-warehouse/tpch.lineitem_orc_def';
alter table lineitem_orc add columns (new_col int);
select count(*) from lineitem_orc where new_col is null;
Verify the final result is 6001215.
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 3
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 09 Jun 2022 07:26:37 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files
Posted by "ttttttz (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18571
to look at the new patch set (#5).
Change subject: IMPALA-11296: Fix infinite loop when reading orc files
......................................................................
IMPALA-11296: Fix infinite loop when reading orc files
When querying an ORC table, selecting only the missing fields of ORC files causes the query to be executed indefinitely. The corresponding execution node will see some resident threads that occupy CPU abnormally. The problem is caused by this: when OrcComplexColumnReader.children_.empty() is true, OrcComplexColumnReader.row_idx_ will remain constant, causing an infinite loop at HdfsOrcScanner::TransferTuples(). We should allow empty 'children_' for original files.
Testing:
- Added a test to test_scanners.py that ensures the query can be executed successfully when selecting only the missing fields of ORC files.
Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
---
M be/src/exec/orc-column-readers.cc
M tests/query_test/test_scanners.py
2 files changed, 26 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/18571/5
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 5
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files
Posted by "ttttttz (Code Review)" <ge...@cloudera.org>.
ttttttz has posted comments on this change. ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files
......................................................................
Patch Set 8:
(6 comments)
> Patch Set 7: Code-Review+1
>
> (1 comment)
>
> The fix LGTM. Just have one more comment in the test.
Hi Quanlong! This adjustment has been made.
http://gerrit.cloudera.org:8080/#/c/18571/3/be/src/exec/orc-column-readers.cc
File be/src/exec/orc-column-readers.cc:
http://gerrit.cloudera.org:8080/#/c/18571/3/be/src/exec/orc-column-readers.cc@410
PS3, Line 410: DC
> nit: we use 2 spaces indention instead of 4 spaces.
Done
http://gerrit.cloudera.org:8080/#/c/18571/3/tests/query_test/test_scanners.py
File tests/query_test/test_scanners.py:
http://gerrit.cloudera.org:8080/#/c/18571/3/tests/query_test/test_scanners.py@1423
PS3, Line 1423:
> nit: "meta" -> "file meta"
Done
http://gerrit.cloudera.org:8080/#/c/18571/3/tests/query_test/test_scanners.py@1423
PS3, Line 1423:
> nit: "field" -> "fields"
Done
http://gerrit.cloudera.org:8080/#/c/18571/3/tests/query_test/test_scanners.py@1424
PS3, Line 1424: @SkipIfIsilon.hive
: @SkipIfLocal.hive
> Don't need these description about the bug behavior. It's fixed by this pat
Done
http://gerrit.cloudera.org:8080/#/c/18571/3/tests/query_test/test_scanners.py@1432
PS3, Line 1432: self.client.execute("refresh %s" % orc_tbl_name)
> Let's also verify the results here.
Done
http://gerrit.cloudera.org:8080/#/c/18571/3/tests/query_test/test_scanners.py@1433
PS3, Line 1433: self.client.execute("alter table %s add columns(f1 int)" % orc_tbl_name)
> Can we also test on a large table? E.g.
Done
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 8
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
Gerrit-Comment-Date: Fri, 10 Jun 2022 08:46:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files.
Posted by "ttttttz (Code Review)" <ge...@cloudera.org>.
ttttttz has posted comments on this change. ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files.
......................................................................
Patch Set 3:
Thanks Quanlong! I have adjusted. Please review it.
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 3
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
Gerrit-Comment-Date: Fri, 10 Jun 2022 02:17:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18571/4/tests/query_test/test_scanners.py
File tests/query_test/test_scanners.py:
http://gerrit.cloudera.org:8080/#/c/18571/4/tests/query_test/test_scanners.py@1438
PS4, Line 1438: n
flake8: E501 line too long (95 > 90 characters)
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 4
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
Gerrit-Comment-Date: Fri, 10 Jun 2022 03:22:17 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files
......................................................................
IMPALA-11296: Fix infinite loop when reading orc files
When querying an ORC table, selecting only the missing fields of ORC
files causes the query to be executed indefinitely. The
corresponding execution node will see some resident threads that
occupy CPU abnormally. The problem is caused by this:
when OrcComplexColumnReader.children_.empty() is true,
OrcComplexColumnReader.row_idx_ will remain constant, causing an
infinite loop at HdfsOrcScanner::TransferTuples().
We should allow empty 'children_' for original files.
Testing:
- Added a test to test_scanners.py that ensures the query can be
executed successfully when selecting only the missing fields of
ORC files.
Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Reviewed-on: http://gerrit.cloudera.org:8080/18571
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Reviewed-by: Zoltan Borok-Nagy <bo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/orc-column-readers.cc
M tests/query_test/test_scanners.py
2 files changed, 32 insertions(+), 0 deletions(-)
Approvals:
Quanlong Huang: Looks good to me, but someone else must approve
Zoltan Borok-Nagy: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 9
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files.
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18571/2/tests/query_test/test_scanners.py
File tests/query_test/test_scanners.py:
http://gerrit.cloudera.org:8080/#/c/18571/2/tests/query_test/test_scanners.py@1424
PS2, Line 1424:
flake8: E501 line too long (94 > 90 characters)
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 2
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 08 Jun 2022 08:50:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files.
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files.
......................................................................
Patch Set 1:
(1 comment)
Thanks for your contribution! Can we add some test cases? The end-to-end tests are driven by pytest, e.g. here is a scanner test file: https://github.com/apache/impala/blob/master/tests/query_test/test_scanners.py
The test cases are in the test files, e.g. testdata/workloads/functional-query/queries/QueryTest/out-of-range-date-orc.test
http://gerrit.cloudera.org:8080/#/c/18571/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18571/1//COMMIT_MSG@8
PS1, Line 8:
Please add the commit message that is exactly long enough to explain what the problem was, and how it was fixed. Each should have 72 or fewer characters if possible.
https://cwiki.apache.org/confluence/display/IMPALA/Contributing+to+Impala
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 1
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 27 May 2022 23:59:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files
......................................................................
Patch Set 7: Code-Review+1
(1 comment)
The fix LGTM. Just have one more comment in the test.
http://gerrit.cloudera.org:8080/#/c/18571/7/tests/query_test/test_scanners.py
File tests/query_test/test_scanners.py:
http://gerrit.cloudera.org:8080/#/c/18571/7/tests/query_test/test_scanners.py@1421
PS7, Line 1421:
This test runs an INSERT statement in Hive. We need to skip it when running the tests in S3, etc. Just like what we do for test_type_conversions(). Please add these annotations:
@SkipIfABFS.hive
@SkipIfADLS.hive
@SkipIfIsilon.hive
@SkipIfLocal.hive
@SkipIfS3.hive
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 7
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
Gerrit-Comment-Date: Fri, 10 Jun 2022 06:43:16 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files
Posted by "ttttttz (Code Review)" <ge...@cloudera.org>.
ttttttz has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files
......................................................................
IMPALA-11296: Fix infinite loop when reading orc files
When querying an ORC table, selecting only the missing fields of ORC
files causes the query to be executed indefinitely. The
corresponding execution node will see some resident threads that
occupy CPU abnormally. The problem is caused by this:
when OrcComplexColumnReader.children_.empty() is true,
OrcComplexColumnReader.row_idx_ will remain constant, causing an
infinite loop at HdfsOrcScanner::TransferTuples().
We should allow empty 'children_' for original files.
Testing:
- Added a test to test_scanners.py that ensures the query can be
executed successfully when selecting only the missing fields of
ORC files.
Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
---
M be/src/exec/orc-column-readers.cc
M tests/query_test/test_scanners.py
2 files changed, 32 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/18571/8
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 8
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files
......................................................................
Patch Set 8: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 8
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
Gerrit-Comment-Date: Fri, 10 Jun 2022 08:07:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR](3.x) IMPALA-11296: Fix infinite loop when reading orc files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18571 )
Change subject: IMPALA-11296: Fix infinite loop when reading orc files
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8215/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 3.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7ecf5e9c94ffcc02d3ca6c2ec8d55a685ec3968
Gerrit-Change-Number: 18571
Gerrit-PatchSet: 8
Gerrit-Owner: ttttttz <24...@qq.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: ttttttz <24...@qq.com>
Gerrit-Comment-Date: Fri, 10 Jun 2022 11:23:36 +0000
Gerrit-HasComments: No