You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Guosmilesmile (via GitHub)" <gi...@apache.org> on 2025/07/24 16:02:55 UTC

[PR] Flink: Add test for adjust the configuration precedence in the dynamic sink [iceberg]

Guosmilesmile opened a new pull request, #13662:
URL: https://github.com/apache/iceberg/pull/13662

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Flink: Add test for adjust the configuration precedence in the dynamic sink [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #13662:
URL: https://github.com/apache/iceberg/pull/13662#discussion_r2238857987


##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/TestDynamicWriter.java:
##########
@@ -180,4 +236,19 @@ private static int getNumDataFiles(Table table) {
     }
     return 0;
   }
+
+  private Map<String, String> properties(DynamicWriter dynamicWriter) {
+    DynFields.BoundField<Map<WriteTarget, TaskWriter<RowData>>> writerField =
+        DynFields.builder().hiddenImpl(dynamicWriter.getClass(), "writers").build(dynamicWriter);
+
+    DynFields.BoundField<FlinkAppenderFactory> appenderField =
+        DynFields.builder()
+            .hiddenImpl(BaseTaskWriter.class, "appenderFactory")
+            .build(writerField.get().values().iterator().next());
+    DynFields.BoundField<Map<String, String>> propsField =
+        DynFields.builder()
+            .hiddenImpl(FlinkAppenderFactory.class, "props")
+            .build(appenderField.get());
+    return propsField.get();
+  }

Review Comment:
   This Reflection magic is hard to read / maintain. Why not add a `getProperties()` on the DynamicWriter?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Flink: Add test for adjust the configuration precedence in the dynamic sink [iceberg]

Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #13662:
URL: https://github.com/apache/iceberg/pull/13662#discussion_r2238866002


##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/TestDynamicWriter.java:
##########
@@ -180,4 +236,19 @@ private static int getNumDataFiles(Table table) {
     }
     return 0;
   }
+
+  private Map<String, String> properties(DynamicWriter dynamicWriter) {
+    DynFields.BoundField<Map<WriteTarget, TaskWriter<RowData>>> writerField =
+        DynFields.builder().hiddenImpl(dynamicWriter.getClass(), "writers").build(dynamicWriter);
+
+    DynFields.BoundField<FlinkAppenderFactory> appenderField =
+        DynFields.builder()
+            .hiddenImpl(BaseTaskWriter.class, "appenderFactory")
+            .build(writerField.get().values().iterator().next());
+    DynFields.BoundField<Map<String, String>> propsField =
+        DynFields.builder()
+            .hiddenImpl(FlinkAppenderFactory.class, "props")
+            .build(appenderField.get());
+    return propsField.get();
+  }

Review Comment:
   Because this property is passed as a parameter to `RowDataTaskWriterFactory` and is a local variable, it cannot be accessed in `DynamicWriter`, so I used the current approach.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Flink: Add test for adjust the configuration precedence in the dynamic sink [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #13662:
URL: https://github.com/apache/iceberg/pull/13662#issuecomment-3133079123

   Merged to main.
   Thanks @Guosmilesmile for the extra tests and @mxm for the review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Flink: Add test for adjust the configuration precedence in the dynamic sink [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary merged PR #13662:
URL: https://github.com/apache/iceberg/pull/13662


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Flink: Add test for adjust the configuration precedence in the dynamic sink [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #13662:
URL: https://github.com/apache/iceberg/pull/13662#discussion_r2238876559


##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/dynamic/TestDynamicWriter.java:
##########
@@ -180,4 +236,19 @@ private static int getNumDataFiles(Table table) {
     }
     return 0;
   }
+
+  private Map<String, String> properties(DynamicWriter dynamicWriter) {
+    DynFields.BoundField<Map<WriteTarget, TaskWriter<RowData>>> writerField =
+        DynFields.builder().hiddenImpl(dynamicWriter.getClass(), "writers").build(dynamicWriter);
+
+    DynFields.BoundField<FlinkAppenderFactory> appenderField =
+        DynFields.builder()
+            .hiddenImpl(BaseTaskWriter.class, "appenderFactory")
+            .build(writerField.get().values().iterator().next());
+    DynFields.BoundField<Map<String, String>> propsField =
+        DynFields.builder()
+            .hiddenImpl(FlinkAppenderFactory.class, "props")
+            .build(appenderField.get());
+    return propsField.get();
+  }

Review Comment:
   You are right, we can't fully test the overrides without peeking into the writers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org