You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/04/21 13:53:34 UTC
[GitHub] [iceberg] southernriver opened a new pull request, #4605: Support insert overwrite table for hive-mr engine.
southernriver opened a new pull request, #4605:
URL: https://github.com/apache/iceberg/pull/4605
Support insert overwrite table for hive-mr engine by setting runtime conf with `set iceberg.mr.write.is.overwrite = true` , otherwise we need to modify hive-module to adapt syntax.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] chenjunjiedada commented on a diff in pull request #4605: Support insert overwrite table for hive-mr engine.
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on code in PR #4605:
URL: https://github.com/apache/iceberg/pull/4605#discussion_r855242537
##########
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -285,16 +288,33 @@ private void commitTable(FileIO io, ExecutorService executor, JobContext jobCont
Collection<DataFile> dataFiles = dataFiles(executor, location, jobContext, io, true);
- if (dataFiles.size() > 0) {
+ boolean isOverwrite = conf.getBoolean(ConfigProperties.IS_OVERWRITE, false);
+
+ if (isOverwrite) {
+ if (!dataFiles.isEmpty()) {
+ ReplacePartitions overwrite = table.newReplacePartitions();
+ dataFiles.forEach(overwrite::addFile);
+ overwrite.commit();
+ LOG.info("Overwrite commit took {} ms for table: {} with {} file(s)", System.currentTimeMillis() - startTime,
+ table, dataFiles.size());
+ } else if (table.spec().isUnpartitioned()) {
+ table.newDelete().deleteFromRowFilter(Expressions.alwaysTrue()).commit();
+ LOG.info("Cleared table contents as part of empty overwrite for unpartitioned table. " +
+ "Commit took {} ms for table: {}", System.currentTimeMillis() - startTime, table);
+ }
Review Comment:
What is the behavior of using empty to overwrite a partitioned table? Should we print some log?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support insert overwrite table for hive-mr engine. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4605:
URL: https://github.com/apache/iceberg/pull/4605#issuecomment-2274584652
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support insert overwrite table for hive-mr engine. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #4605: Support insert overwrite table for hive-mr engine.
URL: https://github.com/apache/iceberg/pull/4605
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support insert overwrite table for hive-mr engine. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4605:
URL: https://github.com/apache/iceberg/pull/4605#issuecomment-2292491777
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] southernriver commented on a diff in pull request #4605: Support insert overwrite table for hive-mr engine.
Posted by GitBox <gi...@apache.org>.
southernriver commented on code in PR #4605:
URL: https://github.com/apache/iceberg/pull/4605#discussion_r855702108
##########
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -285,16 +288,33 @@ private void commitTable(FileIO io, ExecutorService executor, JobContext jobCont
Collection<DataFile> dataFiles = dataFiles(executor, location, jobContext, io, true);
- if (dataFiles.size() > 0) {
+ boolean isOverwrite = conf.getBoolean(ConfigProperties.IS_OVERWRITE, false);
+
+ if (isOverwrite) {
+ if (!dataFiles.isEmpty()) {
+ ReplacePartitions overwrite = table.newReplacePartitions();
+ dataFiles.forEach(overwrite::addFile);
+ overwrite.commit();
+ LOG.info("Overwrite commit took {} ms for table: {} with {} file(s)", System.currentTimeMillis() - startTime,
+ table, dataFiles.size());
+ } else if (table.spec().isUnpartitioned()) {
+ table.newDelete().deleteFromRowFilter(Expressions.alwaysTrue()).commit();
+ LOG.info("Cleared table contents as part of empty overwrite for unpartitioned table. " +
+ "Commit took {} ms for table: {}", System.currentTimeMillis() - startTime, table);
+ }
Review Comment:
Not do anything, I added some log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #4605: Support insert overwrite table for hive-mr engine.
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #4605:
URL: https://github.com/apache/iceberg/pull/4605#issuecomment-1111222452
FWIW this function was developed by @marton-bod for Hive here: [HIVE-25007](https://issues.apache.org/jira/browse/HIVE-25007).
In Hive 4 this is working without the setting of the config with simple Hive queries.
Thanks for porting it @southernriver!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org