You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/07/12 17:15:17 UTC
[GitHub] [iceberg] aokolnychyi opened a new pull request, #5257: Spark 3.3: Expose action classes in SparkActions
aokolnychyi opened a new pull request, #5257:
URL: https://github.com/apache/iceberg/pull/5257
This is a newer version of #5108.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.3: Expose action classes in SparkActions [iceberg]
Posted by "bknbkn (via GitHub)" <gi...@apache.org>.
bknbkn commented on code in PR #5257:
URL: https://github.com/apache/iceberg/pull/5257#discussion_r1436018686
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
Review Comment:
It seem should use ExpireSnapshotsSparkAction in code's 96 line? I also find it in spark 3.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #5257: Spark 3.3: Expose action classes in SparkActions
Posted by GitBox <gi...@apache.org>.
rdblue merged PR #5257:
URL: https://github.com/apache/iceberg/pull/5257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on pull request #5257: Spark 3.3: Expose action classes in SparkActions
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on PR #5257:
URL: https://github.com/apache/iceberg/pull/5257#issuecomment-1182193553
It is easier to review each commit in this PR individually.
We may even want to rebase and merge to keep the commit history.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #5257: Spark 3.3: Expose action classes in SparkActions
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on code in PR #5257:
URL: https://github.com/apache/iceberg/pull/5257#discussion_r919321238
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/actions/BaseDeleteReachableFilesSparkAction.java:
##########
@@ -19,165 +19,21 @@
package org.apache.iceberg.spark.actions;
-import java.util.Iterator;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.atomic.AtomicLong;
-import java.util.function.Consumer;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.TableMetadata;
-import org.apache.iceberg.TableMetadataParser;
-import org.apache.iceberg.actions.BaseDeleteReachableFilesActionResult;
-import org.apache.iceberg.actions.DeleteReachableFiles;
-import org.apache.iceberg.exceptions.NotFoundException;
-import org.apache.iceberg.exceptions.ValidationException;
-import org.apache.iceberg.hadoop.HadoopFileIO;
-import org.apache.iceberg.io.FileIO;
-import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
-import org.apache.iceberg.spark.JobGroupInfo;
-import org.apache.iceberg.util.PropertyUtil;
-import org.apache.iceberg.util.Tasks;
-import org.apache.spark.sql.Dataset;
-import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import static org.apache.iceberg.TableProperties.GC_ENABLED;
-import static org.apache.iceberg.TableProperties.GC_ENABLED_DEFAULT;
/**
- * An implementation of {@link DeleteReachableFiles} that uses metadata tables in Spark
- * to determine which files should be deleted.
+ * An action that deletes reachable files from a given metadata file.
+ *
+ * @deprecated since 0.14.0, will be removed in 1.0.0;
+ * use {@link SparkActions} and {@link DeleteReachableFilesSparkAction} instead.
*/
-@SuppressWarnings("UnnecessaryAnonymousClass")
-public class BaseDeleteReachableFilesSparkAction
Review Comment:
Well, good question. It depends how we treat our Spark libs. If we treat them as independent, I think we can skip the old classes. Let me do that and keep the base classes for 3.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #5257: Spark 3.3: Expose action classes in SparkActions
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5257:
URL: https://github.com/apache/iceberg/pull/5257#discussion_r919312983
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/actions/BaseDeleteReachableFilesSparkAction.java:
##########
@@ -19,165 +19,21 @@
package org.apache.iceberg.spark.actions;
-import java.util.Iterator;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.atomic.AtomicLong;
-import java.util.function.Consumer;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.TableMetadata;
-import org.apache.iceberg.TableMetadataParser;
-import org.apache.iceberg.actions.BaseDeleteReachableFilesActionResult;
-import org.apache.iceberg.actions.DeleteReachableFiles;
-import org.apache.iceberg.exceptions.NotFoundException;
-import org.apache.iceberg.exceptions.ValidationException;
-import org.apache.iceberg.hadoop.HadoopFileIO;
-import org.apache.iceberg.io.FileIO;
-import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
-import org.apache.iceberg.spark.JobGroupInfo;
-import org.apache.iceberg.util.PropertyUtil;
-import org.apache.iceberg.util.Tasks;
-import org.apache.spark.sql.Dataset;
-import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import static org.apache.iceberg.TableProperties.GC_ENABLED;
-import static org.apache.iceberg.TableProperties.GC_ENABLED_DEFAULT;
/**
- * An implementation of {@link DeleteReachableFiles} that uses metadata tables in Spark
- * to determine which files should be deleted.
+ * An action that deletes reachable files from a given metadata file.
+ *
+ * @deprecated since 0.14.0, will be removed in 1.0.0;
+ * use {@link SparkActions} and {@link DeleteReachableFilesSparkAction} instead.
*/
-@SuppressWarnings("UnnecessaryAnonymousClass")
-public class BaseDeleteReachableFilesSparkAction
Review Comment:
@aokolnychyi, Since 3.3 is unreleased, do we need to preserve the base classes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #5257: Spark 3.3: Expose action classes in SparkActions
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on code in PR #5257:
URL: https://github.com/apache/iceberg/pull/5257#discussion_r919275681
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSnapshotUpdateSparkAction.java:
##########
@@ -20,20 +20,17 @@
package org.apache.iceberg.spark.actions;
import java.util.Map;
-import org.apache.iceberg.actions.SnapshotUpdate;
import org.apache.iceberg.relocated.com.google.common.collect.Maps;
import org.apache.spark.sql.SparkSession;
-abstract class BaseSnapshotUpdateSparkAction<ThisT, R>
- extends BaseSparkAction<ThisT, R> implements SnapshotUpdate<ThisT, R> {
+abstract class BaseSnapshotUpdateSparkAction<ThisT> extends BaseSparkAction<ThisT> {
Review Comment:
I had to drop interfaces like `SnapshotUpdate` from parent actions so that `ThisT` can be an action class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org