You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org> on 2024/07/27 22:28:56 UTC
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for count star on jdbc table
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21617
Change subject: IMPALA-13256: Support more than 2G rows for count star on jdbc table
......................................................................
IMPALA-13256: Support more than 2G rows for count star on jdbc table
Function DatabaseAccessor.getTotalNumberOfRecords() is defined with
return data type as int. This causes Impala to thrown exception when
executing count star query for jdbc tables with more than 2G rows.
This patch fixed the issue by changing the function return type as
long. The number of rows in each RowBatch fetched from jdbc data
source cannot exceed 2G.
Testing:
- Passed core-test.
- Manually created a jdbc table jdbc_table with more than 2G rows,
verified that query 'select count(*) from jdbc_table' returned
correct number of rows.
Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
---
M be/src/exec/data-source-scan-node.cc
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
4 files changed, 15 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/21617/2
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for count star on jdbc table
Posted by "Pranav Lodha (Code Review)" <ge...@cloudera.org>.
Pranav Lodha has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for count star on jdbc table
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG@7
PS2, Line 7: count star
nit: Shouldn't it be COUNT(*)
http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG@11
PS2, Line 11: count star
nit: Shouldn't it be COUNT(*)
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Sun, 28 Jul 2024 03:38:35 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Thu, 01 Aug 2024 07:34:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Function DatabaseAccessor.getTotalNumberOfRecords() is defined with
return data type as int. This caused Impala to throw an exception when
executing COUNT(*) query for jdbc tables with more than 2G rows.
This patch fixed the issue by changing the function return type as
long. It also made number of rows in each TRowBatch fetched from jdbc
data source not to exceed 2G.
Testing:
- Passed core-test, and tpcds test for jdbc tables.
- Manually created a jdbc table jdbc_table with more than 2G rows,
verified that query 'select count(*) from jdbc_table' returned
correct number of rows. Detail steps were added in the comments
of IMPALA-13256.
Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
---
M be/src/exec/data-source-scan-node.cc
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
4 files changed, 18 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/21617/3
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
Patch Set 5: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21617/4/be/src/exec/data-source-scan-node.cc
File be/src/exec/data-source-scan-node.cc:
http://gerrit.cloudera.org:8080/#/c/21617/4/be/src/exec/data-source-scan-node.cc@149
PS4, Line 149: DCHECK_LE(input_batch_->rows.num_rows, std::numeric_limits<int32_t>::max());
> fixed as suggested and reran unit-test
Ack
http://gerrit.cloudera.org:8080/#/c/21617/4/fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
File fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java:
http://gerrit.cloudera.org:8080/#/c/21617/4/fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java@220
PS4, Line 220: numRows = totalNumberOfRecords_ - currRow_ <= Integer.MAX_VALUE ?
> We still get the correct results if total number of rows exceed Integer.MAX
Ack
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Wed, 31 Jul 2024 23:21:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
Patch Set 5:
Thanks Michael, Abhishek, Yifan and Pranav.
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Thu, 01 Aug 2024 02:29:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for count star on jdbc table
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for count star on jdbc table
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16587/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Sat, 27 Jul 2024 22:54:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10826/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Thu, 01 Aug 2024 02:27:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16589/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Sun, 28 Jul 2024 20:41:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21617/4/be/src/exec/data-source-scan-node.cc
File be/src/exec/data-source-scan-node.cc:
http://gerrit.cloudera.org:8080/#/c/21617/4/be/src/exec/data-source-scan-node.cc@149
PS4, Line 149: DCHECK_LE(input_batch_->rows.num_rows, 0x7FFFFFFF);
nit: std::numeric_limits<int32_t>::max() would also work, and be more standard.
http://gerrit.cloudera.org:8080/#/c/21617/4/fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
File fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java:
http://gerrit.cloudera.org:8080/#/c/21617/4/fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java@220
PS4, Line 220: numRows = totalNumberOfRecords_ - currRow_ <= Integer.MAX_VALUE ?
Should this be enforced via a limit or something when making the query? It's not clear to me that we still get correct results if we do exceed Integer.MAX_VALUE.
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Wed, 31 Jul 2024 21:31:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for count star on jdbc table
Posted by "Pranav Lodha (Code Review)" <ge...@cloudera.org>.
Pranav Lodha has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for count star on jdbc table
......................................................................
Patch Set 2:
(3 comments)
Thanks Wenzhe!
I've added some minor nits but otherwise the patch looks good.
http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG@10
PS2, Line 10: causes Impala to thrown
nit: caused Impala to throw an
http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG@18
PS2, Line 18: Manually created a jdbc table jdbc_table with more than 2G rows,
: verified that query 'select count(*) from jdbc_table' returned
: correct number of rows.
I think it'd be good if we can add some tests or mention steps/tables for which it failed and works now.
http://gerrit.cloudera.org:8080/#/c/21617/2/be/src/exec/data-source-scan-node.cc
File be/src/exec/data-source-scan-node.cc:
http://gerrit.cloudera.org:8080/#/c/21617/2/be/src/exec/data-source-scan-node.cc@146
PS2, Line 146: DCHECK_LE(input_batch_->rows.num_rows, 0x7FFFFFFF);
nit: Can we add a comment about this check, something like this:
Check to prevent overflow.
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Sun, 28 Jul 2024 03:31:51 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Pranav Lodha (Code Review)" <ge...@cloudera.org>.
Pranav Lodha has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21617/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21617/3//COMMIT_MSG@17
PS3, Line 17: ,
nit: Remove ','
http://gerrit.cloudera.org:8080/#/c/21617/3//COMMIT_MSG@20
PS3, Line 20: Detail
nit: Detailed
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Sun, 28 Jul 2024 19:10:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Function DatabaseAccessor.getTotalNumberOfRecords() is defined with
return data type as int. This caused Impala to throw an exception when
executing COUNT(*) query for jdbc tables with more than 2G rows.
This patch fixed the issue by changing the function return type as
long. It also made number of rows in each TRowBatch fetched from jdbc
data source not to exceed 2G.
Testing:
- Passed core-test and tpcds test for jdbc tables.
- Manually created a jdbc table jdbc_table with more than 2G rows,
verified that query 'select count(*) from jdbc_table' returned
correct number of rows. Detailed steps were added in the comments
of IMPALA-13256.
Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
---
M be/src/exec/data-source-scan-node.cc
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
4 files changed, 19 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/21617/5
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Yifan Zhang (Code Review)" <ge...@cloudera.org>.
Yifan Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
Patch Set 3: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Sun, 28 Jul 2024 13:54:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
Patch Set 3:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG@7
PS2, Line 7: COUNT(*) o
> nit: Shouldn't it be COUNT(*)
fixed
http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG@10
PS2, Line 10: caused Impala to throw
> nit: caused Impala to throw an
fixed
http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG@11
PS2, Line 11: COUNT(*) q
> nit: Shouldn't it be COUNT(*)
fixed
http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG@18
PS2, Line 18: Manually created a jdbc table jdbc_table with more than 2G rows,
: verified that query 'select count(*) from jdbc_table' returned
: correct number of rows.
> I think it'd be good if we can add some tests or mention steps/tables for w
The table is too big and may hit memory issue when creating such big table in runtime. It takes additional disk space in Jenkins and developers' machine if add unit-test. I added detail steps in the comments of IMPALA-13256.
http://gerrit.cloudera.org:8080/#/c/21617/2/be/src/exec/data-source-scan-node.cc
File be/src/exec/data-source-scan-node.cc:
http://gerrit.cloudera.org:8080/#/c/21617/2/be/src/exec/data-source-scan-node.cc@146
PS2, Line 146: // The capacity of output RowBatch is defined as 4
> nit: Can we add a comment about this check, something like this:
Added a comment.
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Sun, 28 Jul 2024 07:03:49 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16588/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Sun, 28 Jul 2024 07:30:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Pranav Lodha (Code Review)" <ge...@cloudera.org>.
Pranav Lodha has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
Patch Set 3: Code-Review+1
(4 comments)
Some minor nits.
http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG@18
PS2, Line 18: Manually created a jdbc table jdbc_table with more than 2G rows,
: verified that query 'select count(*) from jdbc_table' returned
: correct number of rows.
> The table is too big and may hit memory issue when creating such big table
Ack
http://gerrit.cloudera.org:8080/#/c/21617/3/be/src/exec/data-source-scan-node.cc
File be/src/exec/data-source-scan-node.cc:
http://gerrit.cloudera.org:8080/#/c/21617/3/be/src/exec/data-source-scan-node.cc@146
PS3, Line 146: 4 byte
nit: a 4-byte
http://gerrit.cloudera.org:8080/#/c/21617/3/be/src/exec/data-source-scan-node.cc@146
PS3, Line 146: Make sure number
nit: Making sure that the
http://gerrit.cloudera.org:8080/#/c/21617/3/be/src/exec/data-source-scan-node.cc@147
PS3, Line 147: not to exceed
nit: does not exceed
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Sun, 28 Jul 2024 19:08:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Function DatabaseAccessor.getTotalNumberOfRecords() is defined with
return data type as int. This caused Impala to throw an exception when
executing COUNT(*) query for jdbc tables with more than 2G rows.
This patch fixed the issue by changing the function return type as
long. It also made number of rows in each TRowBatch fetched from jdbc
data source not to exceed 2G.
Testing:
- Passed core-test and tpcds test for jdbc tables.
- Manually created a jdbc table jdbc_table with more than 2G rows,
verified that query 'select count(*) from jdbc_table' returned
correct number of rows. Detailed steps were added in the comments
of IMPALA-13256.
Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
---
M be/src/exec/data-source-scan-node.cc
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
4 files changed, 19 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/21617/4
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Thu, 01 Aug 2024 01:53:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Function DatabaseAccessor.getTotalNumberOfRecords() is defined with
return data type as int. This caused Impala to throw an exception when
executing COUNT(*) query for jdbc tables with more than 2G rows.
This patch fixed the issue by changing the function return type as
long. It also made number of rows in each TRowBatch fetched from jdbc
data source not to exceed 2G.
Testing:
- Passed core-test and tpcds test for jdbc tables.
- Manually created a jdbc table jdbc_table with more than 2G rows,
verified that query 'select count(*) from jdbc_table' returned
correct number of rows. Detailed steps were added in the comments
of IMPALA-13256.
Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Reviewed-on: http://gerrit.cloudera.org:8080/21617
Reviewed-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Abhishek Rawat <ar...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/data-source-scan-node.cc
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
4 files changed, 19 insertions(+), 11 deletions(-)
Approvals:
Michael Smith: Looks good to me, but someone else must approve
Abhishek Rawat: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 )
Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16601/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334
Gerrit-Change-Number: 21617
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Wed, 31 Jul 2024 23:46:53 +0000
Gerrit-HasComments: No