You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2025/07/02 12:05:19 UTC
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23110
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
[tools] Add a script to create 300K tables efficiently
Creating a table via Impala or HiveServer2 takes longer than 100ms. The
most efficient way to create bulk of tables might be using the HMS
thrift API directly.
This patch adds a script to create 300K tables. It usually takes 45mins.
Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
---
A testdata/scale_test_metadata/create_300k_tables.py
1 file changed, 79 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/23110/1
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Daniel Becker, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23110
to look at the new patch set (#5).
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
[tools] Add a script to create 300K tables efficiently
Creating a table via Impala or HiveServer2 takes longer than 100ms. The
most efficient way to create bulk of tables might be using the HMS
thrift API directly.
This patch adds a script to create 300K tables. It usually takes 45mins.
Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
---
A testdata/scale_test_metadata/create_300k_tables.py
1 file changed, 86 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/23110/5
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23110 )
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
Patch Set 4:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23110/4/testdata/scale_test_metadata/create_300k_tables.py
File testdata/scale_test_metadata/create_300k_tables.py:
http://gerrit.cloudera.org:8080/#/c/23110/4/testdata/scale_test_metadata/create_300k_tables.py@48
PS4, Line 48: def create_field_schema(col_defs):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/23110/4/testdata/scale_test_metadata/create_300k_tables.py@51
PS4, Line 51: sd.cols = create_field_schema(cols)
flake8: E305 expected 2 blank lines after class or function definition, found 1
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Jul 2025 06:56:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23110 )
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
Patch Set 3:
(3 comments)
Thanks for the feedbacks!
http://gerrit.cloudera.org:8080/#/c/23110/2/testdata/scale_test_metadata/create_300k_tables.py
File testdata/scale_test_metadata/create_300k_tables.py:
http://gerrit.cloudera.org:8080/#/c/23110/2/testdata/scale_test_metadata/create_300k_tables.py@30
PS2, Line 30: 300
> Is it intentional that this is not 300k?
Oops, I set it to 301 for debug. Forgot to reset it back.
http://gerrit.cloudera.org:8080/#/c/23110/2/testdata/scale_test_metadata/create_300k_tables.py@31
PS2, Line 31: DB_NAME = "db_300k"
> The script could also create the DB if it does not exist.
Done
http://gerrit.cloudera.org:8080/#/c/23110/2/testdata/scale_test_metadata/create_300k_tables.py@59
PS2, Line 59: already_exists_tbls = 0
> Shouldn't these come from 'part_cols'?
Oh yeah. Changed this.
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Jul 2025 06:51:51 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Daniel Becker (Code Review)" <ge...@cloudera.org>.
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/23110 )
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
Patch Set 2:
(3 comments)
Thanks Quanlong!
http://gerrit.cloudera.org:8080/#/c/23110/2/testdata/scale_test_metadata/create_300k_tables.py
File testdata/scale_test_metadata/create_300k_tables.py:
http://gerrit.cloudera.org:8080/#/c/23110/2/testdata/scale_test_metadata/create_300k_tables.py@30
PS2, Line 30: 301
Is it intentional that this is not 300k?
http://gerrit.cloudera.org:8080/#/c/23110/2/testdata/scale_test_metadata/create_300k_tables.py@31
PS2, Line 31: DB_NAME = "db_300k"
The script could also create the DB if it does not exist.
http://gerrit.cloudera.org:8080/#/c/23110/2/testdata/scale_test_metadata/create_300k_tables.py@59
PS2, Line 59: partField.name = "part"
Shouldn't these come from 'part_cols'?
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 03 Jul 2025 09:13:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Daniel Becker, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23110
to look at the new patch set (#4).
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
[tools] Add a script to create 300K tables efficiently
Creating a table via Impala or HiveServer2 takes longer than 100ms. The
most efficient way to create bulk of tables might be using the HMS
thrift API directly.
This patch adds a script to create 300K tables. It usually takes 45mins.
Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
---
A testdata/scale_test_metadata/create_300k_tables.py
1 file changed, 84 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/23110/4
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23110 )
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23110/4/testdata/scale_test_metadata/create_300k_tables.py
File testdata/scale_test_metadata/create_300k_tables.py:
http://gerrit.cloudera.org:8080/#/c/23110/4/testdata/scale_test_metadata/create_300k_tables.py@48
PS4, Line 48: tbl = Table()
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/23110/4/testdata/scale_test_metadata/create_300k_tables.py@51
PS4, Line 51: tbl.sd = sd
> flake8: E305 expected 2 blank lines after class or function definition, fou
Done
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Jul 2025 11:19:43 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23110 )
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19746/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Jul 2025 07:16:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23110 )
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19747/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Jul 2025 07:21:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23110 )
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
Patch Set 3:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23110/3/testdata/scale_test_metadata/create_300k_tables.py
File testdata/scale_test_metadata/create_300k_tables.py:
http://gerrit.cloudera.org:8080/#/c/23110/3/testdata/scale_test_metadata/create_300k_tables.py@48
PS3, Line 48: create_field_schema = lambda c: FieldSchema(c[0], c[1], c[2])
flake8: E731 do not assign a lambda expression, use a def
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Jul 2025 06:51:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23110 )
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
Patch Set 1:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23110/1/testdata/scale_test_metadata/create_300k_tables.py
File testdata/scale_test_metadata/create_300k_tables.py:
http://gerrit.cloudera.org:8080/#/c/23110/1/testdata/scale_test_metadata/create_300k_tables.py@39
PS1, Line 39: from impala_thrift_gen.hive_metastore.ttypes import StorageDescriptor
flake8: E402 module level import not at top of file
http://gerrit.cloudera.org:8080/#/c/23110/1/testdata/scale_test_metadata/create_300k_tables.py@72
PS1, Line 72: e
flake8: F841 local variable 'e' is assigned to but never used
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Jul 2025 12:07:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23110 )
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19664/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Jul 2025 12:33:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23110 )
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23110/1/testdata/scale_test_metadata/create_300k_tables.py
File testdata/scale_test_metadata/create_300k_tables.py:
http://gerrit.cloudera.org:8080/#/c/23110/1/testdata/scale_test_metadata/create_300k_tables.py@39
PS1, Line 39: # Prepare a base table object
> flake8: E402 module level import not at top of file
Done
http://gerrit.cloudera.org:8080/#/c/23110/1/testdata/scale_test_metadata/create_300k_tables.py@72
PS1, Line 72: t
> flake8: F841 local variable 'e' is assigned to but never used
Done
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 03 Jul 2025 07:36:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23110
to look at the new patch set (#2).
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
[tools] Add a script to create 300K tables efficiently
Creating a table via Impala or HiveServer2 takes longer than 100ms. The
most efficient way to create bulk of tables might be using the HMS
thrift API directly.
This patch adds a script to create 300K tables. It usually takes 45mins.
Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
---
A testdata/scale_test_metadata/create_300k_tables.py
1 file changed, 84 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/23110/2
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23110 )
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23110/3/testdata/scale_test_metadata/create_300k_tables.py
File testdata/scale_test_metadata/create_300k_tables.py:
http://gerrit.cloudera.org:8080/#/c/23110/3/testdata/scale_test_metadata/create_300k_tables.py@48
PS3, Line 48: def create_field_schema(col_defs):
> flake8: E731 do not assign a lambda expression, use a def
Done
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Jul 2025 06:54:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Daniel Becker, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23110
to look at the new patch set (#3).
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
[tools] Add a script to create 300K tables efficiently
Creating a table via Impala or HiveServer2 takes longer than 100ms. The
most efficient way to create bulk of tables might be using the HMS
thrift API directly.
This patch adds a script to create 300K tables. It usually takes 45mins.
Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
---
A testdata/scale_test_metadata/create_300k_tables.py
1 file changed, 82 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/23110/3
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23110 )
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19751/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Jul 2025 11:45:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [tools] Add a script to create 300K tables efficiently
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23110 )
Change subject: [tools] Add a script to create 300K tables efficiently
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19670/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73bbc4a3ae5bfbfbb508052b4c68b0044f951ac6
Gerrit-Change-Number: 23110
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 03 Jul 2025 08:02:38 +0000
Gerrit-HasComments: No