You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "zabetak (via GitHub)" <gi...@apache.org> on 2026/09/14 15:06:11 UTC
[PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
zabetak opened a new pull request, #6781:
URL: https://github.com/apache/hive/pull/6781
### What changes were proposed in this pull request?
- Add TPCDS planner tests for Iceberg tables backed by S3 container
- Add S3Container using rustfs for tests that need s3a:// access
- Add test driver, init script, and expected output files for all 99 TPCDS queries
### Why are the changes needed?
For catching performance and functionality regressions of the query planner for Iceberg tables.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
```
mvn test -Dtest=TestTPCDSIcebergCliDriver
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4047057864
##########
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
##########
@@ -343,6 +344,33 @@ public TPCDSFormattedCBOConfig() {
}
}
+ public static class TPCDSIcebergS3CliConfig extends AbstractCliConfig {
+ public TPCDSIcebergS3CliConfig() {
+ super(CorePerfCliDriver.class);
+ setQueryDir("ql/src/test/queries/clientpositive/perf");
+ setLogDir("itests/qtest/target/qfile-results/clientpositive/perf/tpcds1tb/iceberg");
+ setResultsDir("ql/src/test/results/clientpositive/perf/tpcds1tb/iceberg");
+
+ setInitScript("q_init_tpcds_iceberg.sql");
+ setCleanupScript("q_test_cleanup_tez.sql");
+ S3Container.BucketSpec bucketSpec = new S3Container.BucketSpec(
+ "dw-team-bucket/data/warehouse/tablespace/external/hive/tpcds_partitioned_iceberg_parquet_10000.db",
+ "https://github.com/zabetak/hive-test-datasets/releases/download/1.1/iceberg_s3_tpcds10tb.zip");
+ setS3Bucket(bucketSpec);
+ setHiveConfDir("data/conf/llap");
+ // Restore hooks to their default values and remove noise from the out file
+ Map<HiveConf.ConfVars, String> conf = new EnumMap<>(HiveConf.ConfVars.class);
+ conf.put(HiveConf.ConfVars.PRE_EXEC_HOOKS, "");
+ conf.put(HiveConf.ConfVars.POST_EXEC_HOOKS, "");
+ // Disable merge join conversion cause it triggers very slow S3 list
+ // operations. Check to reenable later
Review Comment:
Logged https://issues.apache.org/jira/browse/HIVE-30070 and replaced comment with JIRA reference: https://github.com/apache/hive/pull/6781/commits/c0004036ee861dcb668a0b88b665d169058b7363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "rubenada (via GitHub)" <gi...@apache.org>.
rubenada commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4036997790
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestArguments.java:
##########
@@ -209,7 +221,7 @@ public QTestArguments build() {
testArguments.setInitScript(initScript);
testArguments.setCleanupScript(cleanupScript);
testArguments.setWithLlapIo(withLlapIo);
-
+ testArguments.s3BucketSpec = this.s3BucketSpec;
Review Comment:
nit: for consistency with the existing code, this should be done via a setter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "rubenada (via GitHub)" <gi...@apache.org>.
rubenada commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4036866301
##########
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
##########
@@ -343,6 +344,33 @@ public TPCDSFormattedCBOConfig() {
}
}
+ public static class TPCDSIcebergS3CliConfig extends AbstractCliConfig {
+ public TPCDSIcebergS3CliConfig() {
+ super(CorePerfCliDriver.class);
+ setQueryDir("ql/src/test/queries/clientpositive/perf");
+ setLogDir("itests/qtest/target/qfile-results/clientpositive/perf/tpcds1tb/iceberg");
Review Comment:
LogDir and ResultsDir say ".../tpcds1tb/..." (1TB); but below, the bucketSpec is tpcds_partitioned_iceberg_parquet_10000.db and iceberg_s3_tpcds10tb.zip (10TB); is this an inconsistency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4047051467
##########
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
##########
@@ -343,6 +344,33 @@ public TPCDSFormattedCBOConfig() {
}
}
+ public static class TPCDSIcebergS3CliConfig extends AbstractCliConfig {
+ public TPCDSIcebergS3CliConfig() {
+ super(CorePerfCliDriver.class);
+ setQueryDir("ql/src/test/queries/clientpositive/perf");
+ setLogDir("itests/qtest/target/qfile-results/clientpositive/perf/tpcds1tb/iceberg");
Review Comment:
This runs with the 10TB dataset so I fixed the typos: https://github.com/apache/hive/pull/6781/commits/0399773034de3d0a6921c95797d08efff91fc8e7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4047042913
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestArguments.java:
##########
@@ -209,7 +221,7 @@ public QTestArguments build() {
testArguments.setInitScript(initScript);
testArguments.setCleanupScript(cleanupScript);
testArguments.setWithLlapIo(withLlapIo);
-
+ testArguments.s3BucketSpec = this.s3BucketSpec;
Review Comment:
Fixed https://github.com/apache/hive/pull/6781/commits/8840a32d9c3b6bbddec048c3a5f7379e4b3365af
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4047075059
##########
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
##########
@@ -343,6 +344,33 @@ public TPCDSFormattedCBOConfig() {
}
}
+ public static class TPCDSIcebergS3CliConfig extends AbstractCliConfig {
+ public TPCDSIcebergS3CliConfig() {
+ super(CorePerfCliDriver.class);
+ setQueryDir("ql/src/test/queries/clientpositive/perf");
+ setLogDir("itests/qtest/target/qfile-results/clientpositive/perf/tpcds1tb/iceberg");
+ setResultsDir("ql/src/test/results/clientpositive/perf/tpcds1tb/iceberg");
+
+ setInitScript("q_init_tpcds_iceberg.sql");
+ setCleanupScript("q_test_cleanup_tez.sql");
+ S3Container.BucketSpec bucketSpec = new S3Container.BucketSpec(
+ "dw-team-bucket/data/warehouse/tablespace/external/hive/tpcds_partitioned_iceberg_parquet_10000.db",
+ "https://github.com/zabetak/hive-test-datasets/releases/download/1.1/iceberg_s3_tpcds10tb.zip");
Review Comment:
For the moment yes! My personal repo is also used in: standalone-metastore/metastore-server/docker/hive-postgres-tpcds-metastore/Dockerfile
Once I find time, I will try to migrate those to the apache namespace as per discussion: https://lists.apache.org/thread/wdmnhk1tg42vxbpsjz1jc309l2shmr23
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6781:
URL: https://github.com/apache/hive/pull/6781#issuecomment-5667035229
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6781) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6781&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6781&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6781&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6781&metric=new_coverage&view=list)
 [0.6% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6781&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6781)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "rubenada (via GitHub)" <gi...@apache.org>.
rubenada commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4037077393
##########
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
##########
@@ -343,6 +344,33 @@ public TPCDSFormattedCBOConfig() {
}
}
+ public static class TPCDSIcebergS3CliConfig extends AbstractCliConfig {
+ public TPCDSIcebergS3CliConfig() {
+ super(CorePerfCliDriver.class);
+ setQueryDir("ql/src/test/queries/clientpositive/perf");
+ setLogDir("itests/qtest/target/qfile-results/clientpositive/perf/tpcds1tb/iceberg");
+ setResultsDir("ql/src/test/results/clientpositive/perf/tpcds1tb/iceberg");
+
+ setInitScript("q_init_tpcds_iceberg.sql");
+ setCleanupScript("q_test_cleanup_tez.sql");
Review Comment:
just double-checking: is this existing cleanup script sufficient for this new test case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6781:
URL: https://github.com/apache/hive/pull/6781#issuecomment-5678029283
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6781) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6781&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6781&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6781&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6781&metric=new_coverage&view=list)
 [0.6% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6781&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6781)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "rubenada (via GitHub)" <gi...@apache.org>.
rubenada commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4036735549
##########
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
##########
@@ -343,6 +344,33 @@ public TPCDSFormattedCBOConfig() {
}
}
+ public static class TPCDSIcebergS3CliConfig extends AbstractCliConfig {
+ public TPCDSIcebergS3CliConfig() {
+ super(CorePerfCliDriver.class);
+ setQueryDir("ql/src/test/queries/clientpositive/perf");
+ setLogDir("itests/qtest/target/qfile-results/clientpositive/perf/tpcds1tb/iceberg");
+ setResultsDir("ql/src/test/results/clientpositive/perf/tpcds1tb/iceberg");
+
+ setInitScript("q_init_tpcds_iceberg.sql");
+ setCleanupScript("q_test_cleanup_tez.sql");
+ S3Container.BucketSpec bucketSpec = new S3Container.BucketSpec(
+ "dw-team-bucket/data/warehouse/tablespace/external/hive/tpcds_partitioned_iceberg_parquet_10000.db",
+ "https://github.com/zabetak/hive-test-datasets/releases/download/1.1/iceberg_s3_tpcds10tb.zip");
+ setS3Bucket(bucketSpec);
+ setHiveConfDir("data/conf/llap");
+ // Restore hooks to their default values and remove noise from the out file
+ Map<HiveConf.ConfVars, String> conf = new EnumMap<>(HiveConf.ConfVars.class);
+ conf.put(HiveConf.ConfVars.PRE_EXEC_HOOKS, "");
+ conf.put(HiveConf.ConfVars.POST_EXEC_HOOKS, "");
+ // Disable merge join conversion cause it triggers very slow S3 list
+ // operations. Check to reenable later
Review Comment:
nit: "Check to reenable later" maybe worth a TODO here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "rubenada (via GitHub)" <gi...@apache.org>.
rubenada commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4036837938
##########
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
##########
@@ -343,6 +344,33 @@ public TPCDSFormattedCBOConfig() {
}
}
+ public static class TPCDSIcebergS3CliConfig extends AbstractCliConfig {
+ public TPCDSIcebergS3CliConfig() {
+ super(CorePerfCliDriver.class);
+ setQueryDir("ql/src/test/queries/clientpositive/perf");
+ setLogDir("itests/qtest/target/qfile-results/clientpositive/perf/tpcds1tb/iceberg");
+ setResultsDir("ql/src/test/results/clientpositive/perf/tpcds1tb/iceberg");
+
+ setInitScript("q_init_tpcds_iceberg.sql");
+ setCleanupScript("q_test_cleanup_tez.sql");
+ S3Container.BucketSpec bucketSpec = new S3Container.BucketSpec(
+ "dw-team-bucket/data/warehouse/tablespace/external/hive/tpcds_partitioned_iceberg_parquet_10000.db",
+ "https://github.com/zabetak/hive-test-datasets/releases/download/1.1/iceberg_s3_tpcds10tb.zip");
Review Comment:
is it ok to have a personal repo here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4047040645
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -399,6 +407,16 @@ private void createRemoteDirs(HiveConf conf) {
}
}
+ private void setupS3(HiveConf conf) {
+ S3Container.BucketSpec bucket = testArgs.getS3BucketSpec();
+ if(bucket == null) {
Review Comment:
Fixed https://github.com/apache/hive/pull/6781/commits/9ac3296644eaa8f08acb21114715f49af9fac5c6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4142668515
##########
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
##########
@@ -343,6 +344,33 @@ public TPCDSFormattedCBOConfig() {
}
}
+ public static class TPCDSIcebergS3CliConfig extends AbstractCliConfig {
+ public TPCDSIcebergS3CliConfig() {
+ super(CorePerfCliDriver.class);
+ setQueryDir("ql/src/test/queries/clientpositive/perf");
+ setLogDir("itests/qtest/target/qfile-results/clientpositive/perf/tpcds1tb/iceberg");
+ setResultsDir("ql/src/test/results/clientpositive/perf/tpcds1tb/iceberg");
+
+ setInitScript("q_init_tpcds_iceberg.sql");
+ setCleanupScript("q_test_cleanup_tez.sql");
+ S3Container.BucketSpec bucketSpec = new S3Container.BucketSpec(
+ "dw-team-bucket/data/warehouse/tablespace/external/hive/tpcds_partitioned_iceberg_parquet_10000.db",
+ "https://github.com/zabetak/hive-test-datasets/releases/download/1.1/iceberg_s3_tpcds10tb.zip");
Review Comment:
I migrated the datasets to the ASF namespace and added checksum verification on download: https://github.com/apache/hive/pull/6781/commits/dd972f6a420d13836e9b605bd016ffb1ded67220
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6781:
URL: https://github.com/apache/hive/pull/6781#issuecomment-5908767953
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6781) **Quality Gate passed**
Issues
 [69 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6781&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6781&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6781&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6781&metric=new_coverage&view=list)
 [1.7% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6781&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6781)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4047039014
##########
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
##########
@@ -343,6 +344,33 @@ public TPCDSFormattedCBOConfig() {
}
}
+ public static class TPCDSIcebergS3CliConfig extends AbstractCliConfig {
+ public TPCDSIcebergS3CliConfig() {
+ super(CorePerfCliDriver.class);
+ setQueryDir("ql/src/test/queries/clientpositive/perf");
+ setLogDir("itests/qtest/target/qfile-results/clientpositive/perf/tpcds1tb/iceberg");
+ setResultsDir("ql/src/test/results/clientpositive/perf/tpcds1tb/iceberg");
+
+ setInitScript("q_init_tpcds_iceberg.sql");
+ setCleanupScript("q_test_cleanup_tez.sql");
Review Comment:
Fixed https://github.com/apache/hive/pull/6781/commits/e35b69d3601760747f97935995f1a1cb10405f0b
I have the impression that all cleanup scripts are redundant because every test runs in its own JVM so the whole process comes down after the execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4047046974
##########
itests/util/src/main/java/org/apache/hadoop/hive/cli/S3Container.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.cli;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.QTestSystemProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.time.Duration;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipInputStream;
+
+/**
+ * S3 containerized object store for tests that need {@code s3a://} access.
+ */
+public final class S3Container {
+
+ private static final Logger LOG = LoggerFactory.getLogger(S3Container.class);
+ private static final DockerImageName RUSTFS_IMAGE =
+ DockerImageName.parse("rustfs/rustfs:v1.0.0-rc.5-glibc");
+ private static final String ACCESS_KEY = "rustfsadmin";
+ private static final String SECRET_KEY = "rustfsadmin";
+ private static final int S3_PORT = 9000;
+ private static final int DOWNLOAD_CONNECT_TIMEOUT_MS = 30_000;
+ private static final int DOWNLOAD_READ_TIMEOUT_MS = 60_000;
+ private GenericContainer<?> container;
+ private final BucketSpec bucketSpec;
+
+ public S3Container(BucketSpec bucketSpec) {
+ this.bucketSpec = bucketSpec;
+ }
+
+ @SuppressWarnings("resource")
+ public void start() {
+ String bucket = bucketSpec.bucket();
+ container = new GenericContainer<>(RUSTFS_IMAGE)
+ .withExposedPorts(S3_PORT, 9001)
+ .withCreateContainerCmdModifier(cmd ->
+ cmd.withEntrypoint("/bin/sh", "-c",
+ "mkdir -p /data/" + bucket + " && exec /entrypoint.sh rustfs"))
+ .waitingFor(Wait.forListeningPort().withStartupTimeout(Duration.ofMinutes(2)))
+ .withLogConsumer(outputFrame -> LOG.debug(outputFrame.getUtf8String().trim()));
+ container.start();
+ LOG.info("S3 container ready at {}", getS3Endpoint());
+ downloadData();
+ uploadDataToS3();
+ }
+
+ private void uploadDataToS3() {
+ Configuration conf = new Configuration();
+ applyS3Settings(conf);
+ String bucket = bucketSpec.bucket();
+ String keyPrefix = bucketSpec.keyPrefix();
+ try (FileSystem fs = FileSystem.get(URI.create("s3a://" + bucket), conf)) {
+ int count = 0;
+ try (ZipInputStream zis = new ZipInputStream(new FileInputStream(dataPath().toFile()))) {
+ ZipEntry entry;
+ while ((entry = zis.getNextEntry()) != null) {
+ if (entry.isDirectory()) {
+ continue;
+ }
+ String key = keyPrefix.isEmpty() ? entry.getName() : keyPrefix + "/" + entry.getName();
+ Path s3Path = new Path("s3a://" + bucket + "/" + key);
+ try (OutputStream os = fs.create(s3Path, true)) {
+ byte[] buf = new byte[8192];
+ int len;
+ while ((len = zis.read(buf)) != -1) {
+ os.write(buf, 0, len);
+ }
+ }
+ count++;
+ }
+ }
+ LOG.info("Uploaded {} files to s3a://{}", count, bucket);
+ } catch (IOException e) {
+ throw new RuntimeException("Failed to upload data to S3 container", e);
+ }
+ }
+
+ private static final java.nio.file.Path CACHE =
+ Paths.get(QTestSystemProperties.getBuildDir(), "downloads", "S3Container");
+
+ private void downloadData() {
+ try {
+ java.nio.file.Path file = dataPath();
+ if (!Files.exists(file)) {
+ Files.createDirectories(file.getParent());
+ java.nio.file.Path tmp = Files.createTempFile(file.getParent(), "download-", ".tmp");
+ try {
+ LOG.info("Downloading data from {} to {}", bucketSpec.dataUrl, file);
+ FileUtils.copyURLToFile(bucketSpec.dataUrl, tmp.toFile(),
+ DOWNLOAD_CONNECT_TIMEOUT_MS, DOWNLOAD_READ_TIMEOUT_MS);
+ Files.move(tmp, file, StandardCopyOption.ATOMIC_MOVE);
+ } catch (IOException e) {
+ Files.deleteIfExists(tmp);
+ throw e;
+ }
+ }
+ LOG.info("Data from {} are available in {}", bucketSpec.dataUrl, file);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+
+ private java.nio.file.Path dataPath() {
+ return CACHE.resolve(URLEncoder.encode(bucketSpec.dataUrl.toString(), StandardCharsets.UTF_8));
+ }
+
+ public void stop() {
+ if (container != null) {
+ container.stop();
+ container = null;
+ }
+ }
+
+ private String getS3Endpoint() {
+ String host = container.getHost();
+ int mappedPort = container.getMappedPort(S3_PORT);
+ return String.format("http://%s:%d", host, mappedPort);
+ }
+
+ public void applyS3Settings(Configuration conf) {
+ if (container == null) {
+ throw new IllegalStateException("S3 container has not been started");
+ }
+ conf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem");
+ conf.set("fs.s3a.endpoint", getS3Endpoint());
+ conf.set("fs.s3a.access.key", ACCESS_KEY);
+ conf.set("fs.s3a.secret.key", SECRET_KEY);
+ conf.setBoolean("fs.s3a.path.style.access", true);
+ conf.setBoolean("fs.s3a.connection.ssl.enabled", false);
+ conf.set("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem");
+ conf.set("fs.AbstractFileSystem.s3.impl", "org.apache.hadoop.fs.s3a.S3A");
+ }
+
+ public static class BucketSpec {
+ String bucketName;
+ URL dataUrl;
+
+ public BucketSpec(String bucketName, String dataUrl) {
Review Comment:
Good point. I simplified a lot this class in https://github.com/apache/hive/pull/6781/commits/f8c64aa265c1b059fd93fd43d58ecdbd615db2a5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4047009484
##########
itests/util/src/main/java/org/apache/hadoop/hive/cli/S3Container.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.cli;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.QTestSystemProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.time.Duration;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipInputStream;
+
+/**
+ * S3 containerized object store for tests that need {@code s3a://} access.
+ */
+public final class S3Container {
+
+ private static final Logger LOG = LoggerFactory.getLogger(S3Container.class);
+ private static final DockerImageName RUSTFS_IMAGE =
+ DockerImageName.parse("rustfs/rustfs:v1.0.0-rc.5-glibc");
Review Comment:
Fixed https://github.com/apache/hive/pull/6781/commits/b6abc29a34434beca48fc5ced74dd551625ca71a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "rubenada (via GitHub)" <gi...@apache.org>.
rubenada commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4037057578
##########
itests/util/src/main/java/org/apache/hadoop/hive/cli/S3Container.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.cli;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.QTestSystemProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.time.Duration;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipInputStream;
+
+/**
+ * S3 containerized object store for tests that need {@code s3a://} access.
+ */
+public final class S3Container {
+
+ private static final Logger LOG = LoggerFactory.getLogger(S3Container.class);
+ private static final DockerImageName RUSTFS_IMAGE =
+ DockerImageName.parse("rustfs/rustfs:v1.0.0-rc.5-glibc");
Review Comment:
Could we use 1.0.0 (which seems it was released just yesterday: https://github.com/rustfs/rustfs/releases#release-1.0.0) instead of a RC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "rubenada (via GitHub)" <gi...@apache.org>.
rubenada commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4037011698
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -399,6 +407,16 @@ private void createRemoteDirs(HiveConf conf) {
}
}
+ private void setupS3(HiveConf conf) {
+ S3Container.BucketSpec bucket = testArgs.getS3BucketSpec();
+ if(bucket == null) {
Review Comment:
nit: missing space after `if`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #6781:
URL: https://github.com/apache/hive/pull/6781#issuecomment-5694430182
The new tests took ~8 minutes to run in CI: https://ci.hive.apache.org/job/hive-precommit/job/PR-6781/2/testReport/org.apache.hadoop.hive.cli/TestTPCDSIcebergCliDriver/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "rubenada (via GitHub)" <gi...@apache.org>.
rubenada commented on code in PR #6781:
URL: https://github.com/apache/hive/pull/6781#discussion_r4036975938
##########
itests/util/src/main/java/org/apache/hadoop/hive/cli/S3Container.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.cli;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.QTestSystemProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.time.Duration;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipInputStream;
+
+/**
+ * S3 containerized object store for tests that need {@code s3a://} access.
+ */
+public final class S3Container {
+
+ private static final Logger LOG = LoggerFactory.getLogger(S3Container.class);
+ private static final DockerImageName RUSTFS_IMAGE =
+ DockerImageName.parse("rustfs/rustfs:v1.0.0-rc.5-glibc");
+ private static final String ACCESS_KEY = "rustfsadmin";
+ private static final String SECRET_KEY = "rustfsadmin";
+ private static final int S3_PORT = 9000;
+ private static final int DOWNLOAD_CONNECT_TIMEOUT_MS = 30_000;
+ private static final int DOWNLOAD_READ_TIMEOUT_MS = 60_000;
+ private GenericContainer<?> container;
+ private final BucketSpec bucketSpec;
+
+ public S3Container(BucketSpec bucketSpec) {
+ this.bucketSpec = bucketSpec;
+ }
+
+ @SuppressWarnings("resource")
+ public void start() {
+ String bucket = bucketSpec.bucket();
+ container = new GenericContainer<>(RUSTFS_IMAGE)
+ .withExposedPorts(S3_PORT, 9001)
+ .withCreateContainerCmdModifier(cmd ->
+ cmd.withEntrypoint("/bin/sh", "-c",
+ "mkdir -p /data/" + bucket + " && exec /entrypoint.sh rustfs"))
+ .waitingFor(Wait.forListeningPort().withStartupTimeout(Duration.ofMinutes(2)))
+ .withLogConsumer(outputFrame -> LOG.debug(outputFrame.getUtf8String().trim()));
+ container.start();
+ LOG.info("S3 container ready at {}", getS3Endpoint());
+ downloadData();
+ uploadDataToS3();
+ }
+
+ private void uploadDataToS3() {
+ Configuration conf = new Configuration();
+ applyS3Settings(conf);
+ String bucket = bucketSpec.bucket();
+ String keyPrefix = bucketSpec.keyPrefix();
+ try (FileSystem fs = FileSystem.get(URI.create("s3a://" + bucket), conf)) {
+ int count = 0;
+ try (ZipInputStream zis = new ZipInputStream(new FileInputStream(dataPath().toFile()))) {
+ ZipEntry entry;
+ while ((entry = zis.getNextEntry()) != null) {
+ if (entry.isDirectory()) {
+ continue;
+ }
+ String key = keyPrefix.isEmpty() ? entry.getName() : keyPrefix + "/" + entry.getName();
+ Path s3Path = new Path("s3a://" + bucket + "/" + key);
+ try (OutputStream os = fs.create(s3Path, true)) {
+ byte[] buf = new byte[8192];
+ int len;
+ while ((len = zis.read(buf)) != -1) {
+ os.write(buf, 0, len);
+ }
+ }
+ count++;
+ }
+ }
+ LOG.info("Uploaded {} files to s3a://{}", count, bucket);
+ } catch (IOException e) {
+ throw new RuntimeException("Failed to upload data to S3 container", e);
+ }
+ }
+
+ private static final java.nio.file.Path CACHE =
+ Paths.get(QTestSystemProperties.getBuildDir(), "downloads", "S3Container");
+
+ private void downloadData() {
+ try {
+ java.nio.file.Path file = dataPath();
+ if (!Files.exists(file)) {
+ Files.createDirectories(file.getParent());
+ java.nio.file.Path tmp = Files.createTempFile(file.getParent(), "download-", ".tmp");
+ try {
+ LOG.info("Downloading data from {} to {}", bucketSpec.dataUrl, file);
+ FileUtils.copyURLToFile(bucketSpec.dataUrl, tmp.toFile(),
+ DOWNLOAD_CONNECT_TIMEOUT_MS, DOWNLOAD_READ_TIMEOUT_MS);
+ Files.move(tmp, file, StandardCopyOption.ATOMIC_MOVE);
+ } catch (IOException e) {
+ Files.deleteIfExists(tmp);
+ throw e;
+ }
+ }
+ LOG.info("Data from {} are available in {}", bucketSpec.dataUrl, file);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+
+ private java.nio.file.Path dataPath() {
+ return CACHE.resolve(URLEncoder.encode(bucketSpec.dataUrl.toString(), StandardCharsets.UTF_8));
+ }
+
+ public void stop() {
+ if (container != null) {
+ container.stop();
+ container = null;
+ }
+ }
+
+ private String getS3Endpoint() {
+ String host = container.getHost();
+ int mappedPort = container.getMappedPort(S3_PORT);
+ return String.format("http://%s:%d", host, mappedPort);
+ }
+
+ public void applyS3Settings(Configuration conf) {
+ if (container == null) {
+ throw new IllegalStateException("S3 container has not been started");
+ }
+ conf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem");
+ conf.set("fs.s3a.endpoint", getS3Endpoint());
+ conf.set("fs.s3a.access.key", ACCESS_KEY);
+ conf.set("fs.s3a.secret.key", SECRET_KEY);
+ conf.setBoolean("fs.s3a.path.style.access", true);
+ conf.setBoolean("fs.s3a.connection.ssl.enabled", false);
+ conf.set("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem");
+ conf.set("fs.AbstractFileSystem.s3.impl", "org.apache.hadoop.fs.s3a.S3A");
+ }
+
+ public static class BucketSpec {
+ String bucketName;
+ URL dataUrl;
+
+ public BucketSpec(String bucketName, String dataUrl) {
Review Comment:
nit: why not simply passing `bucket` and `keyPrefix` as constructor arguments (and have them as separate final fields), so that we wouldn't need the extra substr logic on the getters (we wouldn't need the getters at all in fact)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30034: Add TPCDS planner tests for Iceberg tables [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6781:
URL: https://github.com/apache/hive/pull/6781#issuecomment-5733661395
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6781) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6781&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6781&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6781&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6781&metric=new_coverage&view=list)
 [0.6% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6781&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6781)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org