You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "bryanck (via GitHub)" <gi...@apache.org> on 2023/04/07 14:37:01 UTC
[GitHub] [iceberg] bryanck opened a new pull request, #7296: Spark: apply rewrite manifest action fix to 3.1,3.2
bryanck opened a new pull request, #7296:
URL: https://github.com/apache/iceberg/pull/7296
This PR applies the changes from https://github.com/apache/iceberg/pull/7263 to Spark v3.2 and v3.1 also. It marks the unused method `SparkUtil.serializableFileIO()` as deprecated, as using this to broadcast a FileIO can lead to unintended consequences, i.e. the underlying S3 client being closed during broadcast removal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on pull request #7296: Spark: apply rewrite manifest action fix to 3.1,3.2
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #7296:
URL: https://github.com/apache/iceberg/pull/7296#issuecomment-1502525973
Merged, thanks @bryanck , and others for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on a diff in pull request #7296: Spark: apply rewrite manifest action fix to 3.1,3.2
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #7296:
URL: https://github.com/apache/iceberg/pull/7296#discussion_r1160890656
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java:
##########
@@ -77,6 +77,8 @@ public class SparkUtil {
private SparkUtil() {}
+ /** @deprecated will be removed in 1.4.0 */
Review Comment:
Great. Maybe we can put a comment on why, and what the user can call.
Actually, I had a question, that I did not have a chance to ask in original pr: https://github.com/apache/iceberg/pull/7263, do we still need to close fileIO, at the end of action? Is that getting handled somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho merged pull request #7296: Spark: apply rewrite manifest action fix to 3.1,3.2
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho merged PR #7296:
URL: https://github.com/apache/iceberg/pull/7296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on a diff in pull request #7296: Spark: apply rewrite manifest action fix to 3.1,3.2
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #7296:
URL: https://github.com/apache/iceberg/pull/7296#discussion_r1160959958
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java:
##########
@@ -77,6 +77,8 @@ public class SparkUtil {
private SparkUtil() {}
+ /** @deprecated will be removed in 1.4.0 */
Review Comment:
Makes sense. I see the SparkActions takes the table, maybe it could be the responsibility of user then to close it, though it is a bit undocumented.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java:
##########
@@ -77,6 +77,8 @@ public class SparkUtil {
private SparkUtil() {}
+ /** @deprecated will be removed in 1.4.0 */
Review Comment:
Makes sense. I see the SparkActions takes the table as argument, maybe it could be the responsibility of user then to close it, though it is a bit undocumented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #7296: Spark: apply rewrite manifest action fix to 3.1,3.2
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #7296:
URL: https://github.com/apache/iceberg/pull/7296#discussion_r1160901851
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java:
##########
@@ -77,6 +77,8 @@ public class SparkUtil {
private SparkUtil() {}
+ /** @deprecated will be removed in 1.4.0 */
Review Comment:
That's a good question and it impacts Spark jobs in general, not just this action, thus this is consistent with Iceberg behavior in general AFAIK. There has been some discussion on handling resource cleanup better, @danielcweeks @rdblue do you have any insights into that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #7296: Spark: apply rewrite manifest action fix to 3.1,3.2
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #7296:
URL: https://github.com/apache/iceberg/pull/7296#discussion_r1160909202
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java:
##########
@@ -77,6 +77,8 @@ public class SparkUtil {
private SparkUtil() {}
+ /** @deprecated will be removed in 1.4.0 */
Review Comment:
I updated this and added comments about the deprecation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org