You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Arnab Karmakar (Code Review)" <ge...@cloudera.org> on 2025/10/14 09:02:00 UTC
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Arnab Karmakar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23536
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
- Add a new "WITH STATS" option to SHOW CREATE TABLE
- When WITH STATS is specified:
* Emit ALTER … ADD PARTITION … ; clauses for existing partitions
* Emit ALTER … SET TBLPROPERTIES for table-level stats (numRows, STATS_GENERATED_VIA_STATS_TASK, etc.)
* Emit ALTER … SET COLUMN STATS for each column (numDVs, numNulls, maxSize, avgSize)
- Update parser, analyzer, and frontend plan to handle the new option
- Add tests to verify correct generation of partition/stat statements
- Ensure backward compatibility: default behavior of SHOW CREATE TABLE remains unchanged
Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
---
M be/src/service/frontend.cc
M be/src/service/frontend.h
M common/thrift/Frontend.thrift
M docs/topics/impala_show.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M tests/metadata/test_show_create_table.py
9 files changed, 432 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/23536/1
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 8:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23536/8/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/8/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@734
PS8, Line 734: Collection<? extends PrunablePartition> partitions = fsTable.getPartitions();
This is an optimization, instead of loading all partitions we just get their metadata.
http://gerrit.cloudera.org:8080/#/c/23536/8/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@756
PS8, Line 756: List<? extends FeFsPartition> sortedPartitions = fsTable.loadPartitions(
Future optimization:
We load the partitions and also get additional info consuming bandwidth. We could build lightweight partitions with empty file descriptor lists.
However, that requires significant refactoring to access HMS client and handle on-demand partition loading
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 8
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 22 Oct 2025 19:21:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 12: Code-Review+2
(1 comment)
LGTM. Carrying Riza's +1. Thanks for your contribution, Arnab!
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@658
PS10, Line 658: out.append("ALTER TABLE ")
: .append(getIdentSql(table.getDb().getName())).append('.')
: .append(getIdentSql(table.getName())).append(' ')
: .append("SET TBLPROPERTIES ")
: .append(propertyMapToSql(allTblProps))
: .append(";\n\n");
: }
:
: // Column stats for non-partition columns
: if (appendColumnStatsStatements(table, out)) {
: out.append("\n");
: }
:
: /
> Apologies, yes you were right, these references do point out htat both tabl
Ack
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 12
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Nov 2025 01:00:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 15: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 15
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Nov 2025 01:28:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 5:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/23536/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23536/4//COMMIT_MSG@11
PS4, Line 11: * Emit ALTER … SET TBLPROPERTIES for table-level stats (numRows,
> This is stale and can be removed.
Done
http://gerrit.cloudera.org:8080/#/c/23536/4/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/4/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@573
PS4, Line 573:
> This is probably not always true/exist, and should be checked against the a
Done
http://gerrit.cloudera.org:8080/#/c/23536/5/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/5/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@626
PS5, Line 626: private static void filterInternalTblPropertiesForStats(
: Map<String, String> props, FeTable table,
: org.apache.hadoop.hive.metastore.api.Table msTbl) {
Is it possible to unify this removal with similar routine done in ToSqlUtils.getCreateTableSql() ?
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 5
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 17 Oct 2025 19:18:53 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23536
to look at the new patch set (#9).
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Add a WITH STATS option to SHOW CREATE TABLE that emits additional statements to recreate table statistics and partitions.
When WITH STATS is specified, Impala emits:
- Base CREATE TABLE statement.
- Table-level ALTER TABLE ... SET TBLPROPERTIES to set properties such as numRows and STATS_GENERATED_VIA_STATS_TASK.
- ALTER TABLE ... SET COLUMN STATS for all non-partition columns (column stats such as numDVs, numNulls, maxSize, avgSize, numTrues, and numFalses).
- For partitioned tables:
- ALTER TABLE ... ADD PARTITION statements to recreate partitions.
- Per-partition ALTER TABLE ... PARTITION (...) SET TBLPROPERTIES to restore partition-level stats like numRows, numFiles, and totalSize.
Partition output is limited by the PARTITION_LIMIT query option (default 1000). Set PARTITION_LIMIT=0 to include all partitions; a warning is emitted if the limit is exceeded.
Parser, analyzer, and frontend plan are updated to support the new option. Tests added to verify table-, column-, and partition-level statements. Default SHOW CREATE TABLE behavior remains unchanged for backward compatibility.
Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
---
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M docs/topics/impala_show.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test
M tests/metadata/test_show_create_table.py
16 files changed, 1,033 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/23536/9
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 9
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 14: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 14
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Nov 2025 01:27:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 7:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23536/3/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/3/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@571
PS3, Line 571: kuduParamsSql, new Pair<>(innerStmt.getSortColumns(),
: innerStmt.getSortingOrder()), properties, innerStmt.getSerdeProperties(),
: innerStmt.isExternal(), innerStmt.ge
> Thanks for your input Quanlong, yes we were a bit confused about the requir
I think either way is OK. But IIRC, we don't have limits like this in the query statement yet. The most similar thing might be query hints. Maybe query option is more flexible, e.g. in the future we can add other options.
http://gerrit.cloudera.org:8080/#/c/23536/6/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/6/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@134
PS6, Line 134: props.remove(StatsSetupConst.DO_NOT_UPDATE_STATS);
:
: // Hide properties that are materialized elsewhere in DDL
: removeHiddenTableProperties(props);
:
: // Table-format specific filtering
: if (table instanceof FeKuduTable) {
> These properties are kept hidden from the user in HIDDEN_TABLE_PROPERTIES b
Don't need to add all properties in HIDDEN_TABLE_PROPERTIES. I think just NUM_ROWS, NUM_FILES and TOTAL_SIZE is enough.
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 7
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 22 Oct 2025 04:02:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23536/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23536/4//COMMIT_MSG@11
PS4, Line 11: * Emit ALTER … ADD PARTITION … ; clauses for existing partitions
This is stale and can be removed.
http://gerrit.cloudera.org:8080/#/c/23536/4/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/4/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@573
PS4, Line 573: 'STATS_GENERATED_VIA_STATS_TASK'='true'
This is probably not always true/exist, and should be checked against the actual key-value in table.getMetaStoreTable().getParameters().
Should this show all table properties instead?
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 4
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Oct 2025 18:34:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23536/5/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/5/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@626
PS5, Line 626: private static void filterInternalTblPropertiesForStats(
: Map<String, String> props, FeTable table,
: org.apache.hadoop.hive.metastore.api.Table msTbl) {
> Is it possible to unify this removal with similar routine done in ToSqlUtil
In that case I'll have to write a common consolidated method for this, and make some changes to the existing code, if thats alright
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 5
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 17 Oct 2025 19:44:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 13: Code-Review+1
The maven resource seems to be available now.
Also, the jenkins seems to be free, I have triggered the dry run now.
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 13
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Nov 2025 06:51:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 14:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12647/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 14
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Nov 2025 09:09:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23536
to look at the new patch set (#12).
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Adds a new WITH STATS option to the SHOW CREATE TABLE statement to
emit additional SQL statements for recreating table statistics and
partitions.
When specified, Impala outputs:
- Base CREATE TABLE statement.
- ALTER TABLE ... SET TBLPROPERTIES for table-level stats.
- ALTER TABLE ... SET COLUMN STATS for all non-partition columns,
restoring column stats.
- For partitioned tables:
- ALTER TABLE ... ADD PARTITION statements to recreate partitions.
- Per-partition ALTER TABLE ... PARTITION (...) SET TBLPROPERTIES
to restore partition-level stats.
Partition output is limited by the PARTITION_LIMIT query option
(default 1000). Setting PARTITION_LIMIT=0 includes all partitions and
emits a warning if the limit is exceeded.
Tests added to verify correctness of emitted statements. Default
behavior of SHOW CREATE TABLE remains unchanged for compatibility.
Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
---
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M docs/topics/impala_show.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test
M tests/metadata/test_show_create_table.py
16 files changed, 1,148 insertions(+), 69 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/23536/12
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 12
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23536/1/tests/metadata/test_show_create_table.py
File tests/metadata/test_show_create_table.py:
http://gerrit.cloudera.org:8080/#/c/23536/1/tests/metadata/test_show_create_table.py@280
PS1, Line 280: ):
> nit: line this up to L279
Didnt get what you're saying, should i make it like this?
def test_kudu_with_stats_table_and_cols(
self, vector, unique_database, with_table_stats):
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Oct 2025 17:19:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23536/12/tests/metadata/test_show_create_table.py
File tests/metadata/test_show_create_table.py:
http://gerrit.cloudera.org:8080/#/c/23536/12/tests/metadata/test_show_create_table.py@218
PS12, Line 218: ("Expected {} statements, got {}".format(
> Another failure is due to this. F-strings is not supported in Python2. Plea
Done
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 13
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Nov 2025 05:28:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 11:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/23536/10/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/23536/10/common/thrift/ImpalaService.thrift@1075
PS10, Line 1075: SHOW_CREATE_TABLE_PARTITION_LIMIT = 198
> The name is too generic that it seems a limit on all statements, including
Done
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@180
PS10, Line 180: // Remove storage handler and internal ids
> It seems this is the only difference between 'createTableProps' and 'alterT
Done
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@494
PS10, Line 494: // Filter properties into two maps: one for CREATE TABLE, one for ALTER TABLE
: Map<String, String> createTableProps = Maps.newLinkedHashMap();
: M
> nit: we can simplify this to only use remove().
Done
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@658
PS10, Line 658: }
: if (numRows >= 0) {
: allTblProps.put("numRows", Long.toString(numRows));
: }
:
: // Add STATS_GENERATED_VIA_STATS_TASK if present in HMS parameters
: if (msTbl != null && msTbl.getParameters() != null) {
: String statsGenerated = msTbl.getParameters().get("STATS_GENERATED_VIA_STATS_TASK");
: if (statsGenerated != null) {
: allTblProps.put("STATS_GENERATED_VIA_STATS_TASK", statsGenerated);
: }
: }
:
: /
> Do we still need these if "numRows" is not removed in filterTblProperties()
FeTable.NUM_ROWS = "numRows" and StatsSetupConst.ROW_COUNT are two different property keys.
These line use table.getNumRows() which is the cached in-memory value from the FeTable interface and falls back to HMS if needed. If the in-memory value is negative (unknown), it tries to get the value from HMS using StatsSetupConst.ROW_COUNT
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@691
PS10, Line 691: out.append(warnings);
: }
:
: return out.toString();
: }
:
: /**
: * Appends ALTER TABLE ... SET COLUMN STATS statements for all non-partition columns
: * that have statistics.
: * @param table The table to generate column stats for
: * @param out StringBuilder to append the statements to
: * @return true if any column stats were appended, false otherwise
: */
: private static boolean appendColumnStatsStatements(FeTable table, StringBuilder out) {
: int numClusterCols = table.getNumClusteringCols();
: boolean hasColumnStats = false;
:
: for (int i = numClusterCols; i < table.getColumns().size(); i++) {
: Column c = table.getColumns().get(i);
: ColumnStats s = c.getStats();
: if (s == null) continue;
: boolean isFixed = c.getType() != null && c.getType().isFixedLengthType();
:
: List<String> kvs = new ArrayList<>();
: // Always include NDV and nulls (may be -1 for unknown)
: kvs.add("'numDVs'='" + s.getNumDistinctValues() + "'");
: kvs.add("'numNulls'='" + s.getNumNulls() + "'");
: // Include size stats only for variable-length types
: if (!isFixed) {
: kvs.add("'maxSize'='" + s.getMaxSize() + "'");
: double avg = s.getAvgSize();
: String avgStr = (Math.rint(avg) == avg) ?
: Long.toString((long) avg) : Double.toString(avg);
: kvs.add("'avgSize'='" + avgStr + "'");
: }
: // Include boolean-specific counts (may be -1 for non-boolean types)
: kvs.add("'numTrues'='" + s.getNumTrues() + "'");
:
> Can we extract these into a method which returns 'hasColumnStats'? The curr
Ive added new helper methods for refactoring the method:
appendColumnStatsStatements()
appendPartitionStatements()
appendAddPartitionStatements()
appendPartitionPropertiesStatements()
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@735
PS10, Line 735:
> I think we can move "numClusterCols > 0" to the check on L731. "numClusterC
Done
http://gerrit.cloudera.org:8080/#/c/23536/10/tests/metadata/test_show_create_table.py
File tests/metadata/test_show_create_table.py:
http://gerrit.cloudera.org:8080/#/c/23536/10/tests/metadata/test_show_create_table.py@261
PS10, Line 261: def __get_partition_properties(self, sql_str):
: """ Extract properties from partition-level SET TBLPROPERTIES statements.
: Handles statements like:
: ALTER TABLE ... PARTITION (p=1) SET TBLPROPERTIES ('key'='value', ...)
> I think the problem is due to missing a space between "TBLPROPERTIES" and "
Ive added a space between TBLPROPERTIES and "(" now so properties_map_regex() is working now.
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 11
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 06 Nov 2025 18:17:53 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23536
to look at the new patch set (#11).
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Adds a new WITH STATS option to the SHOW CREATE TABLE statement to
emit additional SQL statements for recreating table statistics and
partitions.
When specified, Impala outputs:
- Base CREATE TABLE statement.
- ALTER TABLE ... SET TBLPROPERTIES for table-level stats.
- ALTER TABLE ... SET COLUMN STATS for all non-partition columns,
restoring column stats.
- For partitioned tables:
- ALTER TABLE ... ADD PARTITION statements to recreate partitions.
- Per-partition ALTER TABLE ... PARTITION (...) SET TBLPROPERTIES
to restore partition-level stats.
Partition output is limited by the PARTITION_LIMIT query option
(default 1000). Setting PARTITION_LIMIT=0 includes all partitions and
emits a warning if the limit is exceeded.
Tests added to verify correctness of emitted statements. Default
behavior of SHOW CREATE TABLE remains unchanged for compatibility.
Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
---
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M docs/topics/impala_show.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test
M tests/metadata/test_show_create_table.py
16 files changed, 1,177 insertions(+), 74 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/23536/11
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 11
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 7:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23536/3/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/3/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@571
PS3, Line 571: kuduParamsSql, new Pair<>(innerStmt.getSortColumns(),
: innerStmt.getSortingOrder()), properties, innerStmt.getSerdeProperties(),
: innerStmt.isExternal(), innerStmt.ge
> This can have potential scale issue if a table has thousands of partitions,
That's a good point. But sometimes we do need the partition info. The motivation of this JIRA is to ease the steps of reproducing metadata of a table from customers. Currently we have to run SHOW CREATE TABLE + DESCRIBE FORMATTED + SHOW TABLE STATS + SHOW COLUMN STATS + SHOW PARTITIONS on the table. Then in the next round of communication, collect info of some/sampled partitions by DESCRIBE FORMATTED on each of the partitions.
It'd be nice to output some partitions to save the second round of communication. E.g. adding a query option as the limit on number of shown partitions. The default can be 1000. Show a warning message if some partitions are skipped due to exceeding the limit.
http://gerrit.cloudera.org:8080/#/c/23536/6/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/6/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@134
PS6, Line 134: props.remove(StatsSetupConst.DO_NOT_UPDATE_STATS);
:
: // Hide properties that are materialized elsewhere in DDL
: removeHiddenTableProperties(props);
:
: // Table-format specific filtering
: if (table instanceof FeKuduTable) {
> Done
Agree that some stats properties are useful. I see we only add NUM_ROWS so far. Please also add NUM_FILES and TOTAL_SIZE.
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 7
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Oct 2025 10:30:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23536
to look at the new patch set (#10).
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Adds a new WITH STATS option to the SHOW CREATE TABLE statement to
emit additional SQL statements for recreating table statistics and
partitions.
When specified, Impala outputs:
- Base CREATE TABLE statement.
- ALTER TABLE ... SET TBLPROPERTIES for table-level stats.
- ALTER TABLE ... SET COLUMN STATS for all non-partition columns,
restoring column stats.
- For partitioned tables:
- ALTER TABLE ... ADD PARTITION statements to recreate partitions.
- Per-partition ALTER TABLE ... PARTITION (...) SET TBLPROPERTIES
to restore partition-level stats.
Partition output is limited by the PARTITION_LIMIT query option
(default 1000). Setting PARTITION_LIMIT=0 includes all partitions and
emits a warning if the limit is exceeded.
Tests added to verify correctness of emitted statements. Default
behavior of SHOW CREATE TABLE remains unchanged for compatibility.
Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
---
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M docs/topics/impala_show.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test
M tests/metadata/test_show_create_table.py
16 files changed, 1,144 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/23536/10
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 10
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23536/1/common/thrift/Frontend.thrift
File common/thrift/Frontend.thrift:
http://gerrit.cloudera.org:8080/#/c/23536/1/common/thrift/Frontend.thrift@526
PS1, Line 526: 15: optional TShowCreateTableParams show_create_table_params
Changing type of existing thrift field is a breaking change.
Perhaps you can move the with_stats field here into TShowFunctionsParams as the 20th field?
20: optional bool with_stats
http://gerrit.cloudera.org:8080/#/c/23536/1/tests/metadata/test_show_create_table.py
File tests/metadata/test_show_create_table.py:
http://gerrit.cloudera.org:8080/#/c/23536/1/tests/metadata/test_show_create_table.py@280
PS1, Line 280: ):
nit: line this up to L279
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Oct 2025 14:42:37 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23536
to look at the new patch set (#3).
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
- Add a new "WITH STATS" option to SHOW CREATE TABLE
- When WITH STATS is specified:
* Emit ALTER … ADD PARTITION … ; clauses for existing partitions
* Emit ALTER … SET TBLPROPERTIES for table-level stats (numRows, STATS_GENERATED_VIA_STATS_TASK, etc.)
* Emit ALTER … SET COLUMN STATS for each column (numDVs, numNulls, maxSize, avgSize)
- Update parser, analyzer, and frontend plan to handle the new option
- Add tests to verify correct generation of partition/stat statements
- Ensure backward compatibility: default behavior of SHOW CREATE TABLE remains unchanged
Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
---
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M common/thrift/Frontend.thrift
M docs/topics/impala_show.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M tests/metadata/test_show_create_table.py
11 files changed, 431 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/23536/3
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 3
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 12:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@238
PS10, Line 238: protected static String getSortingOrder(Map<String, String> properties) {
> nit: please keep this unchanged.
Done
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@658
PS10, Line 658: out.append("ALTER TABLE ")
: .append(getIdentSql(table.getDb().getName())).append('.')
: .append(getIdentSql(table.getName())).append(' ')
: .append("SET TBLPROPERTIES ")
: .append(propertyMapToSql(allTblProps))
: .append(";\n\n");
: }
:
: // Column stats for non-partition columns
: if (appendColumnStatsStatements(table, out)) {
: out.append("\n");
: }
:
: /
> > FeTable.NUM_ROWS = "numRows" and StatsSetupConst.ROW_COUNT are two differ
Apologies, yes you were right, these references do point out htat both table.getNumRows() and msTbl.getParameters().get(StatsSetupConst.ROW_COUNT) read from the same HMS parameters using the same key ("numRows")
Therefore, there's no case where table.getNumRows() is negative but HMS parameters have "numRows" set, because getNumRows() directly reads from those parameters.
Removing these redundant lines ??
http://gerrit.cloudera.org:8080/#/c/23536/11/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/11/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@469
PS11, Line 469:
> nit: we use 4 spaces indention here so need 2 more spaces
Done
http://gerrit.cloudera.org:8080/#/c/23536/11/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@723
PS11, Line 723:
> nit: add 2 more spaces here
Done
http://gerrit.cloudera.org:8080/#/c/23536/11/fe/src/main/java/org/apache/impala/service/JniFrontend.java
File fe/src/main/java/org/apache/impala/service/JniFrontend.java:
http://gerrit.cloudera.org:8080/#/c/23536/11/fe/src/main/java/org/apache/impala/service/JniFrontend.java@576
PS11, Line 576:
> nit: need 2 more spaces here
Done
http://gerrit.cloudera.org:8080/#/c/23536/11/fe/src/main/java/org/apache/impala/service/JniFrontend.java@578
PS11, Line 578:
> nit: need 2 more spaces here
Done
http://gerrit.cloudera.org:8080/#/c/23536/11/fe/src/main/java/org/apache/impala/service/JniFrontend.java@582
PS11, Line 582:
> nit: need 2 more spaces here
Done
http://gerrit.cloudera.org:8080/#/c/23536/11/tests/metadata/test_show_create_table.py
File tests/metadata/test_show_create_table.py:
http://gerrit.cloudera.org:8080/#/c/23536/11/tests/metadata/test_show_create_table.py@a218
PS11, Line 218:
> nit: I think our code style prefer two blank lines before class definitions
Done
http://gerrit.cloudera.org:8080/#/c/23536/11/tests/metadata/test_show_create_table.py@341
PS11, Line 341: return re.sub(
: properties_map_regex("WITH SERDEPROPERTIES"), "",
: re.sub(properties_map_regex("TBLPROPERTIES"), "", s)).strip()
:
: def __get_properties_map(self, s, properties_map_name):
: retur
> To be simple, I think we don't need changes in __remove_properties_maps().
Done
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 12
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Nov 2025 17:09:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 13: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 13
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Nov 2025 09:07:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 12: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12645/
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 12
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Nov 2025 01:58:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 13:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12646/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 13
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Nov 2025 06:52:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23536
to look at the new patch set (#13).
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Adds a new WITH STATS option to the SHOW CREATE TABLE statement to
emit additional SQL statements for recreating table statistics and
partitions.
When specified, Impala outputs:
- Base CREATE TABLE statement.
- ALTER TABLE ... SET TBLPROPERTIES for table-level stats.
- ALTER TABLE ... SET COLUMN STATS for all non-partition columns,
restoring column stats.
- For partitioned tables:
- ALTER TABLE ... ADD PARTITION statements to recreate partitions.
- Per-partition ALTER TABLE ... PARTITION (...) SET TBLPROPERTIES
to restore partition-level stats.
Partition output is limited by the PARTITION_LIMIT query option
(default 1000). Setting PARTITION_LIMIT=0 includes all partitions and
emits a warning if the limit is exceeded.
Tests added to verify correctness of emitted statements. Default
behavior of SHOW CREATE TABLE remains unchanged for compatibility.
Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
---
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M docs/topics/impala_show.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test
M tests/metadata/test_show_create_table.py
16 files changed, 1,149 insertions(+), 69 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/23536/13
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 13
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 14: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12647/
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 14
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Nov 2025 10:16:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 8:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/23536/8/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/23536/8/be/src/service/client-request-state.cc@575
PS8, Line 575: 1000; // Default value
Can init with this?
query_options().partition_limit;
http://gerrit.cloudera.org:8080/#/c/23536/8/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/23536/8/common/thrift/Query.thrift@813
PS8, Line 813: ;
Please add test with PARTITION_LIMIT=1 over table with more than 1 partition, and validate accordingly.
nit: semicolon is actually not needed.
http://gerrit.cloudera.org:8080/#/c/23536/8/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/8/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@243
PS8, Line 243: private static CreateSqlArtifacts renderCreateTableSql(FeTable table)
: throws CatalogException {
Please move this method just below getCreateTableSql() so the diff to review is smaller.
http://gerrit.cloudera.org:8080/#/c/23536/8/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@845
PS8, Line 845: // Add warning if partitions were skipped
: if (partitionLimit > 0 && totalPartitions > partitionLimit) {
: int skipped = totalPartitions - partitionLimit;
: warnings.append("-- WARNING about partial output\n");
: warnings.append("-- WARNING: Emitted ").append(partitionLimit)
: .append(" of ").append(totalPartitions)
: .append(" partitions (partition_limit=").append(partitionLimit)
: .append("). ").append(skipped).append(" partitions skipped.\n")
: .append("-- To export more partitions, re-run with PARTITION_LIMIT=<n>.\n");
: }
Can this WARNING be validated in the test file?
http://gerrit.cloudera.org:8080/#/c/23536/8/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/23536/8/fe/src/main/java/org/apache/impala/service/Frontend.java@685
PS8, Line 685: 1000
Can be replaced with default value from TQueryOption.
(new TQueryOptions()).getPartition_limit();
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 8
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 23 Oct 2025 04:29:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 9:
(7 comments)
Thanks for continue working on this. I just have few remaining comments.
http://gerrit.cloudera.org:8080/#/c/23536/9//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23536/9//COMMIT_MSG@8
PS9, Line 8:
: Add a WITH STATS option to SHOW CREATE TABLE that emits additional statements to recreate table statistics and partitions.
:
: When WITH STATS is specified, Impala emits:
:
: - Base CREATE TABLE statement.
: - Table-level ALTER TABLE ... SET TBLPROPERTIES to set properties such as numRows and STATS_GENERATED_VIA_STATS_TASK.
: - ALTER TABLE ... SET COLUMN STATS for all non-partition columns (column stats such as numDVs, numNulls, maxSize, avgSize, numTrues, and numFalses).
: - For partitioned tables:
: - ALTER TABLE ... ADD PARTITION statements to recreate partitions.
: - Per-partition ALTER TABLE ... PARTITION (...) SET TBLPROPERTIES to restore partition-level stats like numRows, numFiles, and totalSize.
:
: Partition output is limited by the PARTITION_LIMIT query option (default 1000). Set PARTITION_LIMIT=0 to include all partitions; a warning is emitted if the limit is exceeded.
: Parser, analyzer, and frontend plan are updated to support the new option. Tests added to verify table-, column-, and partition-level statements. Default SHOW CREATE TABLE behavior remains unchanged for backward compatibility.
nit: Please make it so that each commit message lines are 72 chars at most.
See "Fix" section at:
https://cwiki.apache.org/confluence/display/IMPALA/Contributing+to+Impala
http://gerrit.cloudera.org:8080/#/c/23536/8/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/23536/8/be/src/service/client-request-state.cc@575
PS8, Line 575: query_options().partit
> Can init with this?
Done
http://gerrit.cloudera.org:8080/#/c/23536/8/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/23536/8/common/thrift/Query.thrift@813
PS8, Line 813:
> Please add test with PARTITION_LIMIT=1 over table with more than 1 partitio
Done
http://gerrit.cloudera.org:8080/#/c/23536/8/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/8/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@243
PS8, Line 243: * ordering), if 'properties' doesn't contain 'sort.order'.
: */
> Please move this method just below getCreateTableSql() so the diff to revie
Done
http://gerrit.cloudera.org:8080/#/c/23536/8/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@845
PS8, Line 845: if (partitionLimit > 0 && totalPartitions > partitionLimit) {
: int skipped = totalPartitions - partitionLimit;
: warnings.append("-- WARNING about partial output\n");
: warnings.append("-- WARNING: Emitted ").append(partitionLimit)
: .append(" of ").append(totalPartitions)
: .append(" partitions (partition_limit=").append(partitionLimit)
: .append("). ").append(skipped).append(" partitions skipped.\n")
: .append("-- To export more partitions, re-run with PARTITION_LIMIT=<n>.\n");
: }
: }
> Can this WARNING be validated in the test file?
Done
http://gerrit.cloudera.org:8080/#/c/23536/9/testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test
File testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test:
http://gerrit.cloudera.org:8080/#/c/23536/9/testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test@340
PS9, Line 340: ====
: ---- CREATE_TABLE
Please add one test case that test against existing table.
====
---- QUERY
SHOW CREATE TABLE functional.alltypes_date_partition WITH ARGS
http://gerrit.cloudera.org:8080/#/c/23536/9/tests/metadata/test_show_create_table.py
File tests/metadata/test_show_create_table.py:
http://gerrit.cloudera.org:8080/#/c/23536/9/tests/metadata/test_show_create_table.py@186
PS9, Line 186: if not is_paimon:
No need to COMPUTE STATS on existing table.
if not is_paimon and not test_case.existing_table:
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 9
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 23 Oct 2025 23:23:26 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23536/6/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/6/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@134
PS6, Line 134: props.remove(StatsSetupConst.DO_NOT_UPDATE_STATS);
:
: // Hide properties that are materialized elsewhere in DDL
: removeHiddenTableProperties(props);
:
: // Table-format specific filtering
: if (table instanceof FeKuduTable) {
> These removals seems new and do not exist in the old definition of getCreat
Done
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 7
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Oct 2025 03:19:24 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 10:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23536/9/testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test
File testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test:
http://gerrit.cloudera.org:8080/#/c/23536/9/testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test@340
PS9, Line 340: ====
: ---- CREATE_TABLE
> Please add one test case that test against existing table.
Done
http://gerrit.cloudera.org:8080/#/c/23536/9/tests/metadata/test_show_create_table.py
File tests/metadata/test_show_create_table.py:
http://gerrit.cloudera.org:8080/#/c/23536/9/tests/metadata/test_show_create_table.py@186
PS9, Line 186: if not is_paimon
> No need to COMPUTE STATS on existing table.
Done
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 10
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 24 Oct 2025 04:42:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 10: Code-Review+1
(2 comments)
LGTM.
http://gerrit.cloudera.org:8080/#/c/23536/9//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23536/9//COMMIT_MSG@8
PS9, Line 8:
: Adds a new WITH STATS option to the SHOW CREATE TABLE statement to
: emit additional SQL statements for recreating table statistics and
: partitions.
:
: When specified, Impala outputs:
:
: - Base CREATE TABLE statement.
:
: - ALTER TABLE ... SET TBLPROPERTIES for table-level stats.
:
: - ALTER TABLE ... SET COLUMN STATS for all non-partition columns,
: restoring column stats.
:
> nit: Please make it so that each commit message lines are 72 chars at most.
Done
http://gerrit.cloudera.org:8080/#/c/23536/9/tests/metadata/test_show_create_table.py
File tests/metadata/test_show_create_table.py:
http://gerrit.cloudera.org:8080/#/c/23536/9/tests/metadata/test_show_create_table.py@186
PS9, Line 186: if not is_paimon
> Done
Done
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 10
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 24 Oct 2025 21:25:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 11:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@238
PS10, Line 238: protected static String getSortingOrder(Map<String, String> properties) {
nit: please keep this unchanged.
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@658
PS10, Line 658: }
: if (numRows >= 0) {
: allTblProps.put("numRows", Long.toString(numRows));
: }
:
: // Add STATS_GENERATED_VIA_STATS_TASK if present in HMS parameters
: if (msTbl != null && msTbl.getParameters() != null) {
: String statsGenerated = msTbl.getParameters().get("STATS_GENERATED_VIA_STATS_TASK");
: if (statsGenerated != null) {
: allTblProps.put("STATS_GENERATED_VIA_STATS_TASK", statsGenerated);
: }
: }
:
: /
> FeTable.NUM_ROWS = "numRows" and StatsSetupConst.ROW_COUNT are two different property keys.
StatsSetupConst.ROW_COUNT is also "numRows":
https://github.com/apache/hive/blob/6b5febcc559e9f20ebea6fb04679477cf125144f/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java#L103
So I think they are the same key. Or did I miss anything?
On the other side, I think the in-memory row count in Table instances are actually extracted from HMS properties:
https://github.com/apache/impala/blob/826c8cf9b083ed951fbfe649dc9551436efceb7e/fe/src/main/java/org/apache/impala/catalog/Table.java#L458
Is there a case that 'table.getNumRows()' is negative but there is a property of "numRows" set in the HMS properties (msTbl.getParameters())?
http://gerrit.cloudera.org:8080/#/c/23536/11/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/11/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@469
PS11, Line 469:
nit: we use 4 spaces indention here so need 2 more spaces
http://gerrit.cloudera.org:8080/#/c/23536/11/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@723
PS11, Line 723:
nit: add 2 more spaces here
http://gerrit.cloudera.org:8080/#/c/23536/11/fe/src/main/java/org/apache/impala/service/JniFrontend.java
File fe/src/main/java/org/apache/impala/service/JniFrontend.java:
http://gerrit.cloudera.org:8080/#/c/23536/11/fe/src/main/java/org/apache/impala/service/JniFrontend.java@576
PS11, Line 576:
nit: need 2 more spaces here
http://gerrit.cloudera.org:8080/#/c/23536/11/fe/src/main/java/org/apache/impala/service/JniFrontend.java@578
PS11, Line 578:
nit: need 2 more spaces here
http://gerrit.cloudera.org:8080/#/c/23536/11/fe/src/main/java/org/apache/impala/service/JniFrontend.java@582
PS11, Line 582:
nit: need 2 more spaces here
http://gerrit.cloudera.org:8080/#/c/23536/11/tests/metadata/test_show_create_table.py
File tests/metadata/test_show_create_table.py:
http://gerrit.cloudera.org:8080/#/c/23536/11/tests/metadata/test_show_create_table.py@a218
PS11, Line 218:
nit: I think our code style prefer two blank lines before class definitions in Python. So don't need to remove this.
http://gerrit.cloudera.org:8080/#/c/23536/11/tests/metadata/test_show_create_table.py@341
PS11, Line 341: # For partition statements, we need special handling
: if 'PARTITION' in s and 'SET TBLPROPERTIES' in s:
: # Remove: SET TBLPROPERTIES (...) from partition statements
: # Match from SET TBLPROPERTIES ( to the last )
: s = re.sub(r"SET\s+TBLPROPERTIES\s*\(.*\)", "", s, flags=re.DOTALL)
: else:
To be simple, I think we don't need changes in __remove_properties_maps(). Having an additional "SET" at the end is OK if both expected and actual sides have it.
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 11
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Nov 2025 11:16:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 12:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12645/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 12
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Nov 2025 01:01:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 13: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12646/
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 13
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Nov 2025 07:48:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23536
to look at the new patch set (#6).
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
- Add a new "WITH STATS" option to SHOW CREATE TABLE
- When WITH STATS is specified:
* Emit ALTER … SET TBLPROPERTIES for table-level stats (numRows, STATS_GENERATED_VIA_STATS_TASK, etc.)
* Emit ALTER … SET COLUMN STATS for each column (numDVs, numNulls, maxSize, avgSize)
- Update parser, analyzer, and frontend plan to handle the new option
- Add tests to verify correct generation of partition/stat statements
- Ensure backward compatibility: default behavior of SHOW CREATE TABLE remains unchanged
Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
---
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M common/thrift/Frontend.thrift
M docs/topics/impala_show.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test
M tests/metadata/test_show_create_table.py
12 files changed, 769 insertions(+), 184 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/23536/6
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 6
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23536
to look at the new patch set (#2).
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
- Add a new "WITH STATS" option to SHOW CREATE TABLE
- When WITH STATS is specified:
* Emit ALTER … ADD PARTITION … ; clauses for existing partitions
* Emit ALTER … SET TBLPROPERTIES for table-level stats (numRows, STATS_GENERATED_VIA_STATS_TASK, etc.)
* Emit ALTER … SET COLUMN STATS for each column (numDVs, numNulls, maxSize, avgSize)
- Update parser, analyzer, and frontend plan to handle the new option
- Add tests to verify correct generation of partition/stat statements
- Ensure backward compatibility: default behavior of SHOW CREATE TABLE remains unchanged
Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
---
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M common/thrift/Frontend.thrift
M docs/topics/impala_show.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M tests/metadata/test_show_create_table.py
11 files changed, 438 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/23536/2
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 2
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23536/3/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/3/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@571
PS3, Line 571: List<? extends FeFsPartition> partitions = new ArrayList<>(fsTable.loadAllPartitions());
: partitions.sort(HdfsPartition.KV_COMPARATOR);
: for (FeFsPartition p : partitions) {
This can have potential scale issue if a table has thousands of partitions, potentially blowing up CatalogD or local catalog.
I recommend skipping the per-partition ALTER for now, just to be safe.
http://gerrit.cloudera.org:8080/#/c/23536/3/tests/metadata/test_show_create_table.py
File tests/metadata/test_show_create_table.py:
http://gerrit.cloudera.org:8080/#/c/23536/3/tests/metadata/test_show_create_table.py@123
PS3, Line 123: @pytest.mark.parametrize("fmt", ["PARQUET", "TEXTFILE"])
: def test_show_create_table_with_stats_matrix(self, vector, unique_database, cfg, fmt):
Is it possible to follow existing tests in using __run_show_create_table_test_case and .test file?
See ./testdata/workloads/functional-query/queries/QueryTest/show-create-table.test for example.
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 3
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Oct 2025 17:34:35 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23536/1/common/thrift/Frontend.thrift
File common/thrift/Frontend.thrift:
http://gerrit.cloudera.org:8080/#/c/23536/1/common/thrift/Frontend.thrift@526
PS1, Line 526: 17: optional LineageGraph.TLineageGraph lineage_graph
> Changing type of existing thrift field is a breaking change.
Done
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 3
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Oct 2025 17:34:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Adds a new WITH STATS option to the SHOW CREATE TABLE statement to
emit additional SQL statements for recreating table statistics and
partitions.
When specified, Impala outputs:
- Base CREATE TABLE statement.
- ALTER TABLE ... SET TBLPROPERTIES for table-level stats.
- ALTER TABLE ... SET COLUMN STATS for all non-partition columns,
restoring column stats.
- For partitioned tables:
- ALTER TABLE ... ADD PARTITION statements to recreate partitions.
- Per-partition ALTER TABLE ... PARTITION (...) SET TBLPROPERTIES
to restore partition-level stats.
Partition output is limited by the PARTITION_LIMIT query option
(default 1000). Setting PARTITION_LIMIT=0 includes all partitions and
emits a warning if the limit is exceeded.
Tests added to verify correctness of emitted statements. Default
behavior of SHOW CREATE TABLE remains unchanged for compatibility.
Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Reviewed-on: http://gerrit.cloudera.org:8080/23536
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M docs/topics/impala_show.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test
M tests/metadata/test_show_create_table.py
16 files changed, 1,149 insertions(+), 69 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 16
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23536
to look at the new patch set (#5).
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
- Add a new "WITH STATS" option to SHOW CREATE TABLE
- When WITH STATS is specified:
* Emit ALTER … SET TBLPROPERTIES for table-level stats (numRows, STATS_GENERATED_VIA_STATS_TASK, etc.)
* Emit ALTER … SET COLUMN STATS for each column (numDVs, numNulls, maxSize, avgSize)
- Update parser, analyzer, and frontend plan to handle the new option
- Add tests to verify correct generation of partition/stat statements
- Ensure backward compatibility: default behavior of SHOW CREATE TABLE remains unchanged
Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
---
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M common/thrift/Frontend.thrift
M docs/topics/impala_show.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test
M tests/metadata/test_show_create_table.py
12 files changed, 623 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/23536/5
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 5
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 5:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/23536/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23536/4//COMMIT_MSG@11
PS4, Line 11: * Emit ALTER … SET TBLPROPERTIES for table-level stats (numRows,
> This is stale and can be removed.
Done
http://gerrit.cloudera.org:8080/#/c/23536/4/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/4/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@573
PS4, Line 573:
> This is probably not always true/exist, and should be checked against the a
Done
http://gerrit.cloudera.org:8080/#/c/23536/5/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/5/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@42
PS5, Line 42: import org.apache.impala.catalog.FeFsPartition;
Just noticed that Im not using this import anymore, will remove it with next patch
http://gerrit.cloudera.org:8080/#/c/23536/5/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@51
PS5, Line 51: import org.apache.impala.catalog.HdfsPartition;
Just noticed that Im not using this import anymore, will remove it with next patch
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 5
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 17 Oct 2025 17:22:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23536/6/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/6/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@134
PS6, Line 134: props.remove(FeTable.CATALOG_SERVICE_ID);
: props.remove(FeTable.CATALOG_VERSION);
: props.remove(FeTable.LAST_MODIFIED_BY);
: props.remove(FeTable.LAST_MODIFIED_TIME);
: props.remove(FeFsTable.NUM_ERASURE_CODED_FILES);
: props.remove(FeFsTable.NUM_FILES);
: props.remove(FeFsTable.TOTAL_SIZE);
These removals seems new and do not exist in the old definition of getCreateTableSql().
Is this meant to apply only for SHOW CREATE TABLE WITH ARGS only?
Now that I see how many props exist for a table, I wonder if we should limit WITH ARGS only to display NUM_ROWS, NUM_FILES, and TOTAL_SIZE. Planner use that table properties to estimate if table stats does not exist.
But I want to defer on Quanlong for that consideration.
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 6
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 20 Oct 2025 17:34:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 7:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23536/3/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/3/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@571
PS3, Line 571: kuduParamsSql, new Pair<>(innerStmt.getSortColumns(),
: innerStmt.getSortingOrder()), properties, innerStmt.getSerdeProperties(),
: innerStmt.isExternal(), innerStmt.ge
> This can have potential scale issue if a table has thousands of partitions,
Thanks for your input Quanlong, yes we were a bit confused about the requirements of this jira. I'll be using queries of this form now, correct me if there could be any issues.
SHOW CREATE TABLE `sales_data` WITH STATS; (default=1000)
SHOW CREATE TABLE sales_data WITH STATS PARTITION_LIMIT=n;
http://gerrit.cloudera.org:8080/#/c/23536/6/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/6/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@134
PS6, Line 134: props.remove(StatsSetupConst.DO_NOT_UPDATE_STATS);
:
: // Hide properties that are materialized elsewhere in DDL
: removeHiddenTableProperties(props);
:
: // Table-format specific filtering
: if (table instanceof FeKuduTable) {
> Done
These properties are kept hidden from the user in HIDDEN_TABLE_PROPERTIES but I can make them visible if you insist
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 7
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Oct 2025 15:07:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23536/1/tests/metadata/test_show_create_table.py
File tests/metadata/test_show_create_table.py:
http://gerrit.cloudera.org:8080/#/c/23536/1/tests/metadata/test_show_create_table.py@280
PS1, Line 280: ):
> Didnt get what you're saying, should i make it like this?
Yes. No need to line break unnecessarily.
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Oct 2025 17:48:36 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23536/4/testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test
File testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test:
http://gerrit.cloudera.org:8080/#/c/23536/4/testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test@132
PS4, Line 132: TBLPROPERTIES ('deletion-vectors.enabled'='true');
Pushed an extra space by mistake, dont mind it. Will fix it along with other changes in the next patch
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 4
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Oct 2025 17:40:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23536
to look at the new patch set (#4).
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
- Add a new "WITH STATS" option to SHOW CREATE TABLE
- When WITH STATS is specified:
* Emit ALTER … ADD PARTITION … ; clauses for existing partitions
* Emit ALTER … SET TBLPROPERTIES for table-level stats (numRows, STATS_GENERATED_VIA_STATS_TASK, etc.)
* Emit ALTER … SET COLUMN STATS for each column (numDVs, numNulls, maxSize, avgSize)
- Update parser, analyzer, and frontend plan to handle the new option
- Add tests to verify correct generation of partition/stat statements
- Ensure backward compatibility: default behavior of SHOW CREATE TABLE remains unchanged
Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
---
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M common/thrift/Frontend.thrift
M docs/topics/impala_show.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test
M tests/metadata/test_show_create_table.py
12 files changed, 446 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/23536/4
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 4
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23536
to look at the new patch set (#7).
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
- Add a new "WITH STATS" option to SHOW CREATE TABLE
- When WITH STATS is specified:
* Emit ALTER … SET TBLPROPERTIES for table-level stats (numRows, STATS_GENERATED_VIA_STATS_TASK, etc.)
* Emit ALTER … SET COLUMN STATS for each column (numDVs, numNulls, maxSize, avgSize)
- Update parser, analyzer, and frontend plan to handle the new option
- Add tests to verify correct generation of partition/stat statements
- Ensure backward compatibility: default behavior of SHOW CREATE TABLE remains unchanged
Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
---
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M common/thrift/Frontend.thrift
M docs/topics/impala_show.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test
M tests/metadata/test_show_create_table.py
12 files changed, 762 insertions(+), 184 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/23536/7
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 7
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23536
to look at the new patch set (#8).
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
- Add a new "WITH STATS" option to SHOW CREATE TABLE
- When WITH STATS is specified:
* Emit ALTER … SET TBLPROPERTIES for table-level stats (numRows, STATS_GENERATED_VIA_STATS_TASK, etc.)
* Emit ALTER … SET COLUMN STATS for each column (numDVs, numNulls, maxSize, avgSize)
- Update parser, analyzer, and frontend plan to handle the new option
- Add tests to verify correct generation of partition/stat statements
- Ensure backward compatibility: default behavior of SHOW CREATE TABLE remains unchanged
Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
---
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M docs/topics/impala_show.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A testdata/workloads/functional-query/queries/QueryTest/show-create-table-with-stats.test
M tests/metadata/test_show_create_table.py
16 files changed, 994 insertions(+), 176 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/23536/8
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 8
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 12:
(1 comment)
> Patch Set 12: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12645/
There is a failure in building FE. It seems unrelative:
01:13:24 [ERROR] Failed to execute goal on project impala-minimal-s3a-aws-sdk: Could not collect dependencies for project org.apache.impala:impala-minimal-s3a-aws-sdk:jar:5.0.0-SNAPSHOT
01:13:24 [ERROR] Failed to read artifact descriptor for org.wildfly.openssl:wildfly-openssl:jar:2.1.4.ClouderaFinal
01:13:24 [ERROR] Caused by: The following artifacts could not be resolved: org.wildfly.openssl:wildfly-openssl:pom:2.1.4.ClouderaFinal (absent): Could not transfer artifact org.wildfly.openssl:wildfly-openssl:pom:2.1.4.ClouderaFinal from/to impala.cdp.repo (https://native-toolchain-us-west-2.s3.us-west-2.amazonaws.com/build/cdp_components/66846208/maven): status code: 403, reason phrase: Forbidden (403)
http://gerrit.cloudera.org:8080/#/c/23536/12/tests/metadata/test_show_create_table.py
File tests/metadata/test_show_create_table.py:
http://gerrit.cloudera.org:8080/#/c/23536/12/tests/metadata/test_show_create_table.py@218
PS12, Line 218: f"Expected {len(expected_statements)} statements, got {len(create_table_result)}"
Another failure is due to this. F-strings is not supported in Python2. Please use format() instead.
Python 2 compilation failed for tests/metadata/test_show_create_table.py:
Compiling tests/metadata/test_show_create_table.py ...
File "tests/metadata/test_show_create_table.py", line 218
f"Expected {len(expected_statements)} statements, got {len(create_table_result)}"
^
SyntaxError: invalid syntax
https://jenkins.impala.io/job/python-check-ub2004/9972/console
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 12
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Nov 2025 03:11:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 10:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/23536/10/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/23536/10/common/thrift/ImpalaService.thrift@1075
PS10, Line 1075: PARTITION_LIMIT = 198
The name is too generic that it seems a limit on all statements, including SHOW PARTITIONS and SELECT statements. Might be more clear if we use names like SHOW_CREATE_TABLE_PARTITION_LIMIT.
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@180
PS10, Line 180: removeHiddenTableProperties(createTableProps);
It seems this is the only difference between 'createTableProps' and 'alterTableProps'. If so, we can refactor the code to simplify the duplicate operations on them:
* move this to the end of this method and use one map before calling it
* duplicate a new map for 'createTableProps' and invoke removeHiddenTableProperties() on it.
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@494
PS10, Line 494: if (rawProps.containsKey(Table.TBL_PROP_LAST_DDL_TIME)) {
: rawProps.remove(Table.TBL_PROP_LAST_DDL_TIME);
: }
nit: we can simplify this to only use remove().
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@658
PS10, Line 658: // Add numRows if available
: long numRows = table.getNumRows();
: if (numRows < 0) {
: try {
: Map<String, String> params = msTbl != null ? msTbl.getParameters() : null;
: String rc = params != null ? params.get(StatsSetupConst.ROW_COUNT) : null;
: if (rc != null) numRows = Long.parseLong(rc);
: } catch (Exception e) {
: // ignore and keep numRows as unknown
: }
: }
: if (numRows >= 0) {
: allTblProps.put("numRows", Long.toString(numRows));
: }
Do we still need these if "numRows" is not removed in filterTblProperties()?
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@691
PS10, Line 691: // Column stats for non-partition columns: emit a line per column including
: // all applicable stats keys.
: int numClusterCols = table.getNumClusteringCols();
: boolean hasColumnStats = false;
: for (int i = numClusterCols; i < table.getColumns().size(); i++) {
: Column c = table.getColumns().get(i);
: ColumnStats s = c.getStats();
: if (s == null) continue;
: boolean isFixed = c.getType() != null && c.getType().isFixedLengthType();
:
: List<String> kvs = new ArrayList<>();
: // Always include NDV and nulls (may be -1 for unknown)
: kvs.add("'numDVs'='" + s.getNumDistinctValues() + "'");
: kvs.add("'numNulls'='" + s.getNumNulls() + "'");
: // Include size stats only for variable-length types
: if (!isFixed) {
: kvs.add("'maxSize'='" + s.getMaxSize() + "'");
: double avg = s.getAvgSize();
: String avgStr = (Math.rint(avg) == avg) ? Long.toString((long) avg) : Double.toString(avg);
: kvs.add("'avgSize'='" + avgStr + "'");
: }
: // Include boolean-specific counts (may be -1 for non-boolean types)
: kvs.add("'numTrues'='" + s.getNumTrues() + "'");
: kvs.add("'numFalses'='" + s.getNumFalses() + "'");
:
: if (!hasColumnStats) {
: hasColumnStats = true;
: }
: out.append("ALTER TABLE ")
: .append(getIdentSql(table.getDb().getName())).append('.')
: .append(getIdentSql(table.getName())).append(' ')
: .append("SET COLUMN STATS ")
: .append(getIdentSql(c.getName())).append(" (")
: .append(Joiner.on(", ").join(kvs)).append(");\n");
: }
: if (hasColumnStats) {
: out.append("\n");
: }
Can we extract these into a method which returns 'hasColumnStats'? The current getCreateTableWithStatsSql() method is too long (>200 lines). We'd better make it shorter. The following codes adding ADD PARTITION and ALTER PARTITION statements can also be extracted into separate methods.
http://gerrit.cloudera.org:8080/#/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@735
PS10, Line 735: numClusterCols > 0
I think we can move "numClusterCols > 0" to the check on L731. "numClusterCols > 0" means partitioned tables.
http://gerrit.cloudera.org:8080/#/c/23536/10/tests/metadata/test_show_create_table.py
File tests/metadata/test_show_create_table.py:
http://gerrit.cloudera.org:8080/#/c/23536/10/tests/metadata/test_show_create_table.py@261
PS10, Line 261: """ Extract properties from partition-level SET TBLPROPERTIES statements.
: Handles statements like:
: ALTER TABLE ... PARTITION (p=1) SET TBLPROPERTIES ('key'='value', ...)
: The regular properties_map_regex fails here because of nested parentheses.
I think the problem is due to missing a space between "TBLPROPERTIES" and "(" in the ALTER PARTITION statement:
https://gerrit.cloudera.org/c/23536/10/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java#834
Replacing the space after "%s" with "\s*" in properties_map_regex() can solve the problem. We should change it to raw string by the way.
def properties_map_regex(name):
- return "%s \\(([^)]+)\\)" % name
+ return r"%s\s*\(([^)]+)\)" % name
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 10
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Nov 2025 01:51:47 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 15:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12649/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 15
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Nov 2025 01:28:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23536 )
Change subject: IMPALA-13066: Extend SHOW CREATE TABLE to include stats and partitions
......................................................................
Patch Set 15: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/23536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87950ae9d9bb73cb2a435cf5bcad076df1570dc2
Gerrit-Change-Number: 23536
Gerrit-PatchSet: 15
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Nov 2025 06:11:37 +0000
Gerrit-HasComments: No