You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/03/10 23:05:40 UTC
[GitHub] [iceberg] szehon-ho opened a new pull request #4310: Cleanup for Spark DataFrame.overwritePartitions isolation level
szehon-ho opened a new pull request #4310:
URL: https://github.com/apache/iceberg/pull/4310
This makes some minor cleanup and add one more test for #2925
This is from comment https://github.com/apache/iceberg/pull/4293#discussion_r823035860 of splitting these changes from that pr into a separate one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on a change in pull request #4310: Spark: Cleanup for Spark DataFrame.overwritePartitions isolation level
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on a change in pull request #4310:
URL: https://github.com/apache/iceberg/pull/4310#discussion_r826305130
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java
##########
@@ -277,17 +277,15 @@ public void commit(WriterCommitMessage[] messages) {
IsolationLevel isolationLevel = writeConf.isolationLevel();
Long validateFromSnapshotId = writeConf.validateFromSnapshotId();
+ if (isolationLevel != null && validateFromSnapshotId != null) {
+ dynamicOverwrite.validateFromSnapshot(validateFromSnapshotId);
+ }
+
if (isolationLevel == SERIALIZABLE) {
- if (validateFromSnapshotId != null) {
- dynamicOverwrite.validateFromSnapshot(validateFromSnapshotId);
- }
dynamicOverwrite.validateNoConflictingData();
Review comment:
Thanks yea good catch, I was thinking to leave it here to make it more clear in each validation mode what we are validating, and maybe there are future modes where we do not do this as you said. As opposed to the validationSnapshotId which is the basis of all validations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on pull request #4310: Spark 3.2: Cleanup DataFrame overwritePartitions isolation level handling
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on pull request #4310:
URL: https://github.com/apache/iceberg/pull/4310#issuecomment-1067394520
Thanks @kbendick for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on pull request #4310: Spark: Cleanup for Spark DataFrame.overwritePartitions isolation level
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on pull request #4310:
URL: https://github.com/apache/iceberg/pull/4310#issuecomment-1064599588
@aokolnychyi fyi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho merged pull request #4310: Spark 3.2: Cleanup DataFrame overwritePartitions isolation level handling
Posted by GitBox <gi...@apache.org>.
szehon-ho merged pull request #4310:
URL: https://github.com/apache/iceberg/pull/4310
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #4310: Spark: Cleanup for Spark DataFrame.overwritePartitions isolation level
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #4310:
URL: https://github.com/apache/iceberg/pull/4310#discussion_r825387293
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java
##########
@@ -277,17 +277,15 @@ public void commit(WriterCommitMessage[] messages) {
IsolationLevel isolationLevel = writeConf.isolationLevel();
Long validateFromSnapshotId = writeConf.validateFromSnapshotId();
+ if (isolationLevel != null && validateFromSnapshotId != null) {
+ dynamicOverwrite.validateFromSnapshot(validateFromSnapshotId);
+ }
+
if (isolationLevel == SERIALIZABLE) {
- if (validateFromSnapshotId != null) {
- dynamicOverwrite.validateFromSnapshot(validateFromSnapshotId);
- }
dynamicOverwrite.validateNoConflictingData();
Review comment:
Nit: It seems like this one is repeated in both cases. Since there's no existing `else` clause, it's probably not worth reducing the duplication as we might need more `if` statements but thought I'd call it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org