You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "slfan1989 (via GitHub)" <gi...@apache.org> on 2025/08/24 11:07:16 UTC

[PR] Spark4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

slfan1989 opened a new pull request, #13913:
URL: https://github.com/apache/iceberg/pull/13913

   ### Description
   
   This pull request refactors the existing Spark procedures to consistently use `ProcedureInput` for parameter handling. Previously, many stored procedures manually handled parameter extraction from `InternalRow`, leading to inconsistent code across the project. By adopting `ProcedureInput`, we improve the code uniformity and ensure that all stored procedures now use a unified approach for parameter handling.
   
   ### Changes:
   - Replaced manual parameter extraction with `ProcedureInput` in relevant stored procedure code.
   - Simplified parameter validation and conversion by using `ProcedureInput` methods like `asString`, `asLong`, `asInt`, etc.
   - Enhanced readability and maintainability by consolidating parameter handling logic into a centralized class.
   
   ### Benefits:
   - **Consistency**: All stored procedures now handle parameters in a consistent and standardized way.
   - **Readability**: Code is more readable and easier to maintain by leveraging `ProcedureInput`'s built-in functions.
   - **Reduced Duplication**: Avoids repeated parameter extraction and conversion logic throughout the project.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2363977766


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,13 +118,14 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, false);
+    Long[] snapshotIds = input.asLongArray(SNAPSHOT_IDS_PARAM, null);
+    boolean cleanExpiredMetadata = input.asBoolean(CLEAN_EXPIRED_METADATA_PARAM, false);

Review Comment:
   I would be in favor of using `boolean` instead of `Boolean` but we do actually have cases where we want to not pass an argument to the underlying core API if the user didn't provide any input and thus rely on whatever the default behavior of the core API is



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2369648366


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,13 +118,14 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, false);
+    Long[] snapshotIds = input.asLongArray(SNAPSHOT_IDS_PARAM, null);
+    boolean cleanExpiredMetadata = input.asBoolean(CLEAN_EXPIRED_METADATA_PARAM, false);

Review Comment:
   > but we do actually have cases where we want to not pass an argument to the underlying core API if the user didn't provide any input and thus rely on whatever the default behavior of the core API is
   
   From my understanding, I believe @nastra wanted us to keep original boxed boolean to convey user intention, as unset default of null implies not apply such flag to underlying spark action. Now we switch to false as primitive boolean, it will always pass such flag to action. 
   
   Sorry I missed this initially, let's switch to `Boolean` just like before this change and keep the minimal necessary change to adopt `ProcedureInput`. Maybe also worth a comment to convey the message as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2363858431


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RemoveOrphanFilesProcedure.java:
##########
@@ -105,58 +124,39 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    String location = args.isNullAt(2) ? null : args.getString(2);
-    boolean dryRun = args.isNullAt(3) ? false : args.getBoolean(3);
-    Integer maxConcurrentDeletes = args.isNullAt(4) ? null : args.getInt(4);
-    String fileListView = args.isNullAt(5) ? null : args.getString(5);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    String location = input.asString(LOCATION_PARAM, null);
+    boolean dryRun = input.asBoolean(DRY_RUN_PARAM, false);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    String fileListView = input.asString(FILE_LIST_VIEW_PARAM, null);
 
     Preconditions.checkArgument(
         maxConcurrentDeletes == null || maxConcurrentDeletes > 0,
         "max_concurrent_deletes should have value > 0, value: %s",
         maxConcurrentDeletes);
 
-    Map<String, String> equalSchemes = Maps.newHashMap();
-    if (!args.isNullAt(6)) {
-      args.getMap(6)
-          .foreach(
-              DataTypes.StringType,
-              DataTypes.StringType,
-              (k, v) -> {
-                equalSchemes.put(k.toString(), v.toString());
-                return BoxedUnit.UNIT;
-              });
-    }
-
-    Map<String, String> equalAuthorities = Maps.newHashMap();
-    if (!args.isNullAt(7)) {
-      args.getMap(7)
-          .foreach(
-              DataTypes.StringType,
-              DataTypes.StringType,
-              (k, v) -> {
-                equalAuthorities.put(k.toString(), v.toString());
-                return BoxedUnit.UNIT;
-              });
-    }
+    Map<String, String> equalSchemes = input.asStringMap(EQUAL_SCHEMES_PARAM, ImmutableMap.of());
+    Map<String, String> equalAuthorities =
+        input.asStringMap(EQUAL_AUTHORITIES_PARAM, ImmutableMap.of());
 
-    PrefixMismatchMode prefixMismatchMode =
-        args.isNullAt(8) ? null : PrefixMismatchMode.fromString(args.getString(8));
+    PrefixMismatchMode prefixMismatchMode = input.asPrefixMismatchMode(PREFIX_MISMATCH_MODE_PARAM);
 
-    boolean prefixListing = args.isNullAt(9) ? false : args.getBoolean(9);
+    boolean prefixListing = input.asBoolean(PREFIX_LISTING_PARAM, false);
 
+    Long finalOlderThanMillis = olderThanMillis;

Review Comment:
   I also suggested this earlier in https://github.com/apache/iceberg/pull/13913#discussion_r2317242962



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2371034604


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ProcedureInput.java:
##########
@@ -92,6 +109,22 @@ public Long asLong(ProcedureParameter param, Long defaultValue) {
     return args.isNullAt(ordinal) ? defaultValue : (Long) args.getLong(ordinal);
   }
 
+  public long[] asLongArray(ProcedureParameter param, Long[] defaultValue) {
+    validateParamType(param, DataTypes.createArrayType(DataTypes.LongType));
+    Long[] source =
+        array(param, (array, ordinal) -> array.getLong(ordinal), Long.class, defaultValue);
+
+    if (source == null) {
+      return null;

Review Comment:
   Thanks for your comment! The `defaultValue` has already been used in the array method:
   
   ```
   Long[] source =
       array(param, (array, ordinal) -> array.getLong(ordinal), Long.class, defaultValue);
   ```
   
   Therefore, if ` source` returns null, it means that `defaultValue` is also `null`, and in this case returning `null` is reasonable. Currently, the type of `defaultValue` is `Long[]`. If we were to return `defaultValue` directly here, we would need to introduce additional redundant logic just to satisfy Java syntax.
   
   I think returning null is a more appropriate approach since it keeps the code simpler. Is it fine if I handle it this way?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2371316098


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteManifestsProcedure.java:
##########
@@ -89,9 +92,10 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Boolean useCaching = args.isNullAt(1) ? null : args.getBoolean(1);
-    Integer specId = args.isNullAt(2) ? null : args.getInt(2);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Boolean useCaching = input.asBoolean(USE_CACHING_PARAM, false);

Review Comment:
   the default should be `null` and not `false` here in order to maintain the original behavior



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2362940532


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,13 +118,14 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, false);
+    Long[] snapshotIds = input.asLongArray(SNAPSHOT_IDS_PARAM, null);
+    boolean cleanExpiredMetadata = input.asBoolean(CLEAN_EXPIRED_METADATA_PARAM, false);

Review Comment:
   this changes semantics. previously it was expecting a `Boolean` and now it gets a `boolean`. If the parameter wasn't defined by the caller, then we would rely on the API's default value (instead of passing true/false). That's also why the null checks further below existed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#issuecomment-3330902640

   @nastra Thanks for reviewing and merging the code! @dramaticlly Thanks for taking the time to review it!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2370834980


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,13 +118,14 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, false);
+    Long[] snapshotIds = input.asLongArray(SNAPSHOT_IDS_PARAM, null);
+    boolean cleanExpiredMetadata = input.asBoolean(CLEAN_EXPIRED_METADATA_PARAM, false);

Review Comment:
   @nastra @dramaticlly Thank you for helping review the code! I have already reverted this part of the code back to Boolean.



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,13 +118,14 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, false);
+    Long[] snapshotIds = input.asLongArray(SNAPSHOT_IDS_PARAM, null);
+    boolean cleanExpiredMetadata = input.asBoolean(CLEAN_EXPIRED_METADATA_PARAM, false);

Review Comment:
   @nastra @dramaticlly Thank you for helping review the code! I have already reverted this part of the code back to `Boolean`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2369648366


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,13 +118,14 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, false);
+    Long[] snapshotIds = input.asLongArray(SNAPSHOT_IDS_PARAM, null);
+    boolean cleanExpiredMetadata = input.asBoolean(CLEAN_EXPIRED_METADATA_PARAM, false);

Review Comment:
   > but we do actually have cases where we want to not pass an argument to the underlying core API if the user didn't provide any input and thus rely on whatever the default behavior of the core API is
   
   From my understanding, I believe @nastra wanted us to keep original boxed boolean to convey user intention, as unset default of null implies not apply such flag to underlying spark action. Now we switch to false as primitive boolean, it will always pass such flag to action. Sorry I missed this initially, let's switch to `Boolean` just like before this change and keep the minimal necessary change to adopt `ProcedureInput`. Maybe also worth a comment to convey the message as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#issuecomment-3305539109

   > @nastra Could you please help review this PR? Thank you very much! This PR unifies input parameter parsing using ProcedureInput to enhance consistency and maintainability.
   > 
   > cc: @dramaticlly
   
   @nastra Could you kindly review this PR? In this PR, we have standardized the input parameter parsing for all stored procedures, and going forward, the parameter parsing for all stored procedures will be consistent. Thank you very much!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2364993020


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ProcedureInput.java:
##########
@@ -80,6 +82,22 @@ public Integer asInt(ProcedureParameter param, Integer defaultValue) {
     return args.isNullAt(ordinal) ? defaultValue : (Integer) args.getInt(ordinal);
   }
 
+  public long asTimestampLong(ProcedureParameter param) {
+    Long value = asTimestampLong(param, null);
+    Preconditions.checkArgument(value != null, "Parameter '%s' is not set", param.name());
+    return value;
+  }
+
+  public Long asTimestampLong(ProcedureParameter param, Long defaultValue) {

Review Comment:
   Thank you for reviewing the code. I will make the adjustments based on your suggestions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2371316716


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteManifestsProcedure.java:
##########
@@ -89,9 +92,10 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Boolean useCaching = args.isNullAt(1) ? null : args.getBoolean(1);
-    Integer specId = args.isNullAt(2) ? null : args.getInt(2);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Boolean useCaching = input.asBoolean(USE_CACHING_PARAM, false);

Review Comment:
   ```suggestion
       Boolean useCaching = input.asBoolean(USE_CACHING_PARAM, null);
   ```
   the default should be `null` and not `false` here in order to maintain the original behavior



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteManifestsProcedure.java:
##########
@@ -89,9 +92,10 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Boolean useCaching = args.isNullAt(1) ? null : args.getBoolean(1);
-    Integer specId = args.isNullAt(2) ? null : args.getInt(2);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Boolean useCaching = input.asBoolean(USE_CACHING_PARAM, false);

Review Comment:
   the default should be `null` and not `false` here in order to maintain the original behavior



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2325620659


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,26 +118,28 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    Boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, null);

Review Comment:
   I saw you updated the `cleanExpiredMetadata` to boolean but there's also `streamResult` here, feel we kind do the same, just change from `if (streamResult != null)` to `if (streamResult)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#issuecomment-3251638167

   > > @dramaticlly Thank you very much for reviewing the code and for your valuable suggestions! I have adopted most of your suggestions. Regarding the suggestion to modify the field types, my consideration is that this PR mainly focuses on improving input parameter parsing. Changing the field types might require adjustments to the execution logic, so in order to maintain consistency with the original author's setup, I would prefer not to make this change. Do you think this is acceptable?
   > 
   > Thanks @slfan1989 , generally I think separate changes into multiple pulls are acceptable. However for Spark, we generally want to keep the supported Spark versions in sync, so we likely will need a back port PR to have same change for Spark 3.4 and 3.5. Many PRs with back port can fanout quickly
   > 
   > These suggested change from Boolean to boolean shall be relative straightforward, and our existing unit tests shall catch the problem with right coverage. Otherwise there's bigger problem in our procedure tests. Please let me know if this changes your mind.
   
   @dramaticlly Thank you for your explanation! I believe it makes sense. I will try to improve the code based on your suggestions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2362940532


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,13 +118,14 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, false);
+    Long[] snapshotIds = input.asLongArray(SNAPSHOT_IDS_PARAM, null);
+    boolean cleanExpiredMetadata = input.asBoolean(CLEAN_EXPIRED_METADATA_PARAM, false);

Review Comment:
   generally speaking, we need to make sure that we don't change return types as this changes semantics. previously it was expecting a `Boolean` and now it gets a `boolean`. If the parameter wasn't defined by the caller, then we would rely on the API's default value (instead of passing true/false). That's also why the null checks further below existed.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2366814132


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,13 +118,14 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, false);
+    Long[] snapshotIds = input.asLongArray(SNAPSHOT_IDS_PARAM, null);
+    boolean cleanExpiredMetadata = input.asBoolean(CLEAN_EXPIRED_METADATA_PARAM, false);

Review Comment:
   we need to switch back to using `Boolean` if it was `Boolean` originally



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2317559815


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,26 +118,28 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    Boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, null);
+    Long[] snapshotIds = input.asLongArray(SNAPSHOT_IDS_PARAM, null);
+    Boolean cleanExpiredMetadata = input.asBoolean(CLEAN_EXPIRED_METADATA_PARAM, null);
 
     Preconditions.checkArgument(
         maxConcurrentDeletes == null || maxConcurrentDeletes > 0,
         "max_concurrent_deletes should have value > 0, value: %s",
         maxConcurrentDeletes);
 
+    Long finalOlderThanMillis = olderThanMillis;

Review Comment:
   You are right, we can remove the finalOlderThanMillis variable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2366043571


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ProcedureInput.java:
##########
@@ -119,6 +142,13 @@ public String[] asStringArray(ProcedureParameter param, String[] defaultValue) {
         defaultValue);
   }
 
+  public DeleteOrphanFiles.PrefixMismatchMode asPrefixMismatchMode(ProcedureParameter param) {

Review Comment:
   This part of the code has already been improved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2366033923


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,13 +118,14 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, false);
+    Long[] snapshotIds = input.asLongArray(SNAPSHOT_IDS_PARAM, null);

Review Comment:
   Thank you for your suggestion. I have adjusted the return value to `long[]` to maintain consistency with the original usage.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2363850099


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,13 +118,14 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, false);
+    Long[] snapshotIds = input.asLongArray(SNAPSHOT_IDS_PARAM, null);
+    boolean cleanExpiredMetadata = input.asBoolean(CLEAN_EXPIRED_METADATA_PARAM, false);

Review Comment:
   Good catch on the boolean handling. I actually suggested @slfan1989 convert these to primitive booleans because all the procedures here use boolean parameters as explicit enablement flags, where null defaults to disabled. 
   
   This gives us a chance to clean up the `if $Boolean != null` pattern below when setting the corresponding flags.
   Since we typically backport changes from the latest Spark version to older ones, making this change now should save us some backport work later, even though it means a bit more review effort upfront. Our unit tests should catch any behavioral changes from the semantic shift.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2317568128


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,26 +118,28 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    Boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, null);

Review Comment:
   Thank you for your suggestion! I find your opinion reasonable. However, after reviewing the code before the changes, I noticed that the original author used the Boolean type. The goal of our PR is to adjust the way input parameters are parsed, so I would prefer to keep the variable type as originally intended by the author in this case. Do you think this approach is acceptable?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#issuecomment-3247452948

   @dramaticlly Thank you very much for reviewing the code and for your valuable suggestions! I have adopted most of your suggestions. Regarding the suggestion to modify the field types, my consideration is that this PR mainly focuses on improving input parameter parsing. Changing the field types might require adjustments to the execution logic, so in order to maintain consistency with the original author's setup, I would prefer not to make this change. Do you think this is acceptable?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2317235312


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,26 +118,28 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());

Review Comment:
   ```suggestion
       Identifier tableIdent = input.ident(TABLE_PARAM);
   ```



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,26 +118,28 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    Boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, null);

Review Comment:
   why don't we have 
   
   ```java
       boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, false);
   ```
   
   and only set if it's true below? Same apply to cleanExpiredMetadata



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/SetCurrentSnapshotProcedure.java:
##########
@@ -87,9 +90,10 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long snapshotId = args.isNullAt(1) ? null : args.getLong(1);
-    String ref = args.isNullAt(2) ? null : args.getString(2);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());

Review Comment:
   ```suggestion
       Identifier tableIdent = input.ident(TABLE_PARAM);
   ```



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/CherrypickSnapshotProcedure.java:
##########
@@ -83,8 +85,10 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    long snapshotId = args.getLong(1);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());

Review Comment:
   ```suggestion
       Identifier tableIdent = input.ident(TABLE_PARAM);
   ```



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,26 +118,28 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    Boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, null);
+    Long[] snapshotIds = input.asLongArray(SNAPSHOT_IDS_PARAM, null);
+    Boolean cleanExpiredMetadata = input.asBoolean(CLEAN_EXPIRED_METADATA_PARAM, null);
 
     Preconditions.checkArgument(
         maxConcurrentDeletes == null || maxConcurrentDeletes > 0,
         "max_concurrent_deletes should have value > 0, value: %s",
         maxConcurrentDeletes);
 
+    Long finalOlderThanMillis = olderThanMillis;

Review Comment:
   do we need this finalOlderThanMillis? It seems we can run `TestExpireSnapshotsProcedure` successfully without this 



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/FastForwardBranchProcedure.java:
##########
@@ -75,9 +78,11 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    String from = args.getString(1);
-    String to = args.getString(2);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());

Review Comment:
   ```suggestion
       Identifier tableIdent = input.ident(TABLE_PARAM);
   ```



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RemoveOrphanFilesProcedure.java:
##########
@@ -105,58 +124,39 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    String location = args.isNullAt(2) ? null : args.getString(2);
-    boolean dryRun = args.isNullAt(3) ? false : args.getBoolean(3);
-    Integer maxConcurrentDeletes = args.isNullAt(4) ? null : args.getInt(4);
-    String fileListView = args.isNullAt(5) ? null : args.getString(5);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());

Review Comment:
   ```suggestion
       Identifier tableIdent = input.ident(TABLE_PARAM);
   ```



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RollbackToTimestampProcedure.java:
##########
@@ -83,9 +84,10 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());

Review Comment:
   ```suggestion
       Identifier tableIdent = input.ident(TABLE_PARAM);
   ```



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/PublishChangesProcedure.java:
##########
@@ -87,8 +89,10 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    String wapId = args.getString(1);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());

Review Comment:
   ```suggestion
       Identifier tableIdent = input.ident(TABLE_PARAM);
   ```



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ProcedureInput.java:
##########
@@ -119,6 +148,15 @@ public String[] asStringArray(ProcedureParameter param, String[] defaultValue) {
         defaultValue);
   }
 
+  public DeleteOrphanFiles.PrefixMismatchMode asPrefixMismatchMode(ProcedureParameter param) {
+    String modeAsString = asString(param, null);
+    DeleteOrphanFiles.PrefixMismatchMode prefixMismatchMode =
+        (modeAsString == null)
+            ? null
+            : DeleteOrphanFiles.PrefixMismatchMode.fromString(modeAsString);
+    return prefixMismatchMode;

Review Comment:
   ```suggestion
         return (modeAsString == null)
             ? null
             : DeleteOrphanFiles.PrefixMismatchMode.fromString(modeAsString);
   ```



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteManifestsProcedure.java:
##########
@@ -89,9 +92,10 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Boolean useCaching = args.isNullAt(1) ? null : args.getBoolean(1);
-    Integer specId = args.isNullAt(2) ? null : args.getInt(2);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());
+    Boolean useCaching = input.asBoolean(USE_CACHING_PARAM, false);

Review Comment:
   let's use primitive boolean as well



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RegisterTableProcedure.java:
##########
@@ -81,9 +83,11 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
     TableIdentifier tableName =
-        Spark3Util.identifierToTableIdentifier(toIdentifier(args.getString(0), "table"));
-    String metadataFile = args.getString(1);
+        Spark3Util.identifierToTableIdentifier(
+            toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name()));

Review Comment:
   ```suggestion
   input.ident(TABLE_PARAM);
   ```



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RollbackToSnapshotProcedure.java:
##########
@@ -82,8 +84,9 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    long snapshotId = args.getLong(1);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());

Review Comment:
   ```suggestion
       Identifier tableIdent = input.ident(TABLE_PARAM);
   ```



##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ProcedureInput.java:
##########
@@ -92,6 +110,17 @@ public Long asLong(ProcedureParameter param, Long defaultValue) {
     return args.isNullAt(ordinal) ? defaultValue : (Long) args.getLong(ordinal);
   }
 
+  public Long[] asLongArray(ProcedureParameter param) {
+    Long[] value = asLongArray(param, null);
+    Preconditions.checkArgument(value != null, "Parameter '%s' is not set", param.name());
+    return value;
+  }

Review Comment:
   looks like this is not needed for this change, let's add later when the needs come



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2371316716


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteManifestsProcedure.java:
##########
@@ -89,9 +92,10 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Boolean useCaching = args.isNullAt(1) ? null : args.getBoolean(1);
-    Integer specId = args.isNullAt(2) ? null : args.getInt(2);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Boolean useCaching = input.asBoolean(USE_CACHING_PARAM, false);

Review Comment:
   ```suggestion
       Boolean useCaching = input.asBoolean(USE_CACHING_PARAM, null);
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2362958512


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteManifestsProcedure.java:
##########
@@ -89,18 +92,17 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Boolean useCaching = args.isNullAt(1) ? null : args.getBoolean(1);
-    Integer specId = args.isNullAt(2) ? null : args.getInt(2);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    boolean useCaching = input.asBoolean(USE_CACHING_PARAM, false);

Review Comment:
   semantics are being changed here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2326638023


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,26 +118,28 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    Boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, null);

Review Comment:
   Thank you for your suggestions! I have optimized the related code.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#issuecomment-3250000680

   > @dramaticlly Thank you very much for reviewing the code and for your valuable suggestions! I have adopted most of your suggestions. Regarding the suggestion to modify the field types, my consideration is that this PR mainly focuses on improving input parameter parsing. Changing the field types might require adjustments to the execution logic, so in order to maintain consistency with the original author's setup, I would prefer not to make this change. Do you think this is acceptable?
   
   Thanks @slfan1989 , generally I think separate changes into multiple pulls are acceptable. However for Spark, we generally want to keep the supported Spark versions in sync, so we likely will need a back port PR to have same change for Spark 3.4 and 3.5, so it can fan out pretty quickly. 
   
   These suggested change from Boolean to boolean shall be relative straightforward, and our existing unit tests shall catch the problem with right coverage. Otherwise there's bigger problem in our procedure tests. Please let me know if this changes your mind. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2362933716


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,13 +118,14 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, false);
+    Long[] snapshotIds = input.asLongArray(SNAPSHOT_IDS_PARAM, null);

Review Comment:
   why not make those return `long[]` instead of `Long[]`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2362948804


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ProcedureInput.java:
##########
@@ -119,6 +142,13 @@ public String[] asStringArray(ProcedureParameter param, String[] defaultValue) {
         defaultValue);
   }
 
+  public DeleteOrphanFiles.PrefixMismatchMode asPrefixMismatchMode(ProcedureParameter param) {

Review Comment:
   I don't think we should put this method here as it's only used in a single place



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2362922140


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ProcedureInput.java:
##########
@@ -80,6 +82,22 @@ public Integer asInt(ProcedureParameter param, Integer defaultValue) {
     return args.isNullAt(ordinal) ? defaultValue : (Integer) args.getInt(ordinal);
   }
 
+  public long asTimestampLong(ProcedureParameter param) {

Review Comment:
   ```suggestion
     public long asTimestampMillis(ProcedureParameter param) {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2362956437


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RemoveOrphanFilesProcedure.java:
##########
@@ -105,58 +124,39 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    String location = args.isNullAt(2) ? null : args.getString(2);
-    boolean dryRun = args.isNullAt(3) ? false : args.getBoolean(3);
-    Integer maxConcurrentDeletes = args.isNullAt(4) ? null : args.getInt(4);
-    String fileListView = args.isNullAt(5) ? null : args.getString(5);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    String location = input.asString(LOCATION_PARAM, null);
+    boolean dryRun = input.asBoolean(DRY_RUN_PARAM, false);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    String fileListView = input.asString(FILE_LIST_VIEW_PARAM, null);
 
     Preconditions.checkArgument(
         maxConcurrentDeletes == null || maxConcurrentDeletes > 0,
         "max_concurrent_deletes should have value > 0, value: %s",
         maxConcurrentDeletes);
 
-    Map<String, String> equalSchemes = Maps.newHashMap();
-    if (!args.isNullAt(6)) {
-      args.getMap(6)
-          .foreach(
-              DataTypes.StringType,
-              DataTypes.StringType,
-              (k, v) -> {
-                equalSchemes.put(k.toString(), v.toString());
-                return BoxedUnit.UNIT;
-              });
-    }
-
-    Map<String, String> equalAuthorities = Maps.newHashMap();
-    if (!args.isNullAt(7)) {
-      args.getMap(7)
-          .foreach(
-              DataTypes.StringType,
-              DataTypes.StringType,
-              (k, v) -> {
-                equalAuthorities.put(k.toString(), v.toString());
-                return BoxedUnit.UNIT;
-              });
-    }
+    Map<String, String> equalSchemes = input.asStringMap(EQUAL_SCHEMES_PARAM, ImmutableMap.of());
+    Map<String, String> equalAuthorities =
+        input.asStringMap(EQUAL_AUTHORITIES_PARAM, ImmutableMap.of());
 
-    PrefixMismatchMode prefixMismatchMode =
-        args.isNullAt(8) ? null : PrefixMismatchMode.fromString(args.getString(8));
+    PrefixMismatchMode prefixMismatchMode = input.asPrefixMismatchMode(PREFIX_MISMATCH_MODE_PARAM);
 
-    boolean prefixListing = args.isNullAt(9) ? false : args.getBoolean(9);
+    boolean prefixListing = input.asBoolean(PREFIX_LISTING_PARAM, false);
 
+    Long finalOlderThanMillis = olderThanMillis;

Review Comment:
   why is this one needed? It doesn't seem that `olderThanMillis` is being modified here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2362920244


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ProcedureInput.java:
##########
@@ -80,6 +82,22 @@ public Integer asInt(ProcedureParameter param, Integer defaultValue) {
     return args.isNullAt(ordinal) ? defaultValue : (Integer) args.getInt(ordinal);
   }
 
+  public long asTimestampLong(ProcedureParameter param) {
+    Long value = asTimestampLong(param, null);
+    Preconditions.checkArgument(value != null, "Parameter '%s' is not set", param.name());
+    return value;
+  }
+
+  public Long asTimestampLong(ProcedureParameter param, Long defaultValue) {

Review Comment:
   this should be called `asTimestampMillis`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2366046903


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ExpireSnapshotsProcedure.java:
##########
@@ -104,13 +118,14 @@ public ProcedureParameter[] parameters() {
   @Override
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Long olderThanMillis = args.isNullAt(1) ? null : DateTimeUtil.microsToMillis(args.getLong(1));
-    Integer retainLastNum = args.isNullAt(2) ? null : args.getInt(2);
-    Integer maxConcurrentDeletes = args.isNullAt(3) ? null : args.getInt(3);
-    Boolean streamResult = args.isNullAt(4) ? null : args.getBoolean(4);
-    long[] snapshotIds = args.isNullAt(5) ? null : args.getArray(5).toLongArray();
-    Boolean cleanExpiredMetadata = args.isNullAt(6) ? null : args.getBoolean(6);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Long olderThanMillis = input.asTimestampLong(OLDER_THAN_PARAM, null);
+    Integer retainLastNum = input.asInt(RETAIN_LAST_PARAM, null);
+    Integer maxConcurrentDeletes = input.asInt(MAX_CONCURRENT_DELETES_PARAM, null);
+    boolean streamResult = input.asBoolean(STREAM_RESULTS_PARAM, false);
+    Long[] snapshotIds = input.asLongArray(SNAPSHOT_IDS_PARAM, null);
+    boolean cleanExpiredMetadata = input.asBoolean(CLEAN_EXPIRED_METADATA_PARAM, false);

Review Comment:
   Thank you for the information! From my perspective, changing it to boolean is feasible. The unit tests have all passed, and the results are as expected. Shall we go ahead and accept this improvement?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2366815837


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/ProcedureInput.java:
##########
@@ -92,6 +109,22 @@ public Long asLong(ProcedureParameter param, Long defaultValue) {
     return args.isNullAt(ordinal) ? defaultValue : (Long) args.getLong(ordinal);
   }
 
+  public long[] asLongArray(ProcedureParameter param, Long[] defaultValue) {
+    validateParamType(param, DataTypes.createArrayType(DataTypes.LongType));
+    Long[] source =
+        array(param, (array, ordinal) -> array.getLong(ordinal), Long.class, defaultValue);
+
+    if (source == null) {
+      return null;

Review Comment:
   should this rather be returning the `defaultValue`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2371327275


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteManifestsProcedure.java:
##########
@@ -89,9 +92,10 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    Boolean useCaching = args.isNullAt(1) ? null : args.getBoolean(1);
-    Integer specId = args.isNullAt(2) ? null : args.getInt(2);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+    Identifier tableIdent = input.ident(TABLE_PARAM);
+    Boolean useCaching = input.asBoolean(USE_CACHING_PARAM, false);

Review Comment:
   Thank you for your suggestions! I have made the modifications.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra merged PR #13913:
URL: https://github.com/apache/iceberg/pull/13913


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.0: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. [iceberg]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13913:
URL: https://github.com/apache/iceberg/pull/13913#discussion_r2317558726


##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/CherrypickSnapshotProcedure.java:
##########
@@ -83,8 +85,10 @@ public ProcedureParameter[] parameters() {
 
   @Override
   public Iterator<Scan> call(InternalRow args) {
-    Identifier tableIdent = toIdentifier(args.getString(0), PARAMETERS[0].name());
-    long snapshotId = args.getLong(1);
+    ProcedureInput input = new ProcedureInput(spark(), tableCatalog(), PARAMETERS, args);
+
+    Identifier tableIdent = toIdentifier(input.asString(TABLE_PARAM), TABLE_PARAM.name());

Review Comment:
   Thank you for your suggestion! I have made changes to this part of the code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org