You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/01/28 09:45:50 UTC
[GitHub] [iceberg] hililiwei opened a new pull request #3998: Flink: support overwrite configuration in the flink sink action
hililiwei opened a new pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998
Supports overwriting table configuration, such as file size, when creating Flink sink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r819776672
##########
File path: flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java
##########
@@ -326,4 +327,51 @@ public void testTwoSinksInDisjointedDAG() throws Exception {
Assert.assertEquals("rightTable", rightTable.currentSnapshot().summary().get("direction"));
}
+ @Test
+ public void testOverrideWriteConfigWithUnknownDistributionMode() {
+ AssertHelpers.assertThrows("Override write configuration.",
+ RuntimeException.class, "No enum constant org.apache.iceberg.DistributionMode.UNRECOGNIZED",
+ () -> {
+ Map<String, String> newProps = Maps.newHashMap();
+ newProps.put(TableProperties.WRITE_DISTRIBUTION_MODE, "UNRECOGNIZED");
+
+ List<Row> rows = createRows("");
+ DataStream<Row> dataStream = env.addSource(createBoundedSource(rows), ROW_TYPE_INFO);
+
+ FlinkSink.forRow(dataStream, SimpleDataUtil.FLINK_SCHEMA)
+ .table(table)
+ .tableLoader(tableLoader)
+ .writeParallelism(parallelism)
+ .properties(newProps)
+ .append();
+
+ // Execute the program.
+ env.execute("Test Iceberg DataStream.");
Review comment:
This is the only line that needs to be in the assertion, right? You can think of this as a try/catch where you don't want to have an overly broad try block, or else you might not catch the exception you want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3998: Flink: support overwrite configuration in the flink sink action
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r798794636
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -168,6 +170,11 @@ public Builder table(Table newTable) {
return this;
}
+ public Builder properties(Map<String, String> newProps) {
Review comment:
let's add a Javadoc to explain the motivation/usage of this API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r810467791
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -442,6 +455,16 @@ private String operatorName(String suffix) {
throw new RuntimeException("Unrecognized write.distribution-mode: " + writeMode);
}
}
+
+ private void buildProperties() {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r837014168
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -339,15 +348,29 @@ public Builder setSnapshotProperty(String property, String value) {
*/
@Deprecated
public DataStreamSink<RowData> build() {
- return chainIcebergOperators();
+ return append();
}
/**
* Append the iceberg sink operators to write records to iceberg table.
*
* @return {@link DataStreamSink} for sink.
*/
- public DataStreamSink<Void> append() {
+ @SuppressWarnings("rawtypes")
+ public DataStreamSink append() {
+ Preconditions.checkNotNull(tableLoader, "Table loader shouldn't be null");
+
+ if (table == null) {
+ tableLoader.open();
+ try (TableLoader loader = tableLoader) {
+ this.table = loader.loadTable();
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to load iceberg table from table loader: " + tableLoader, e);
+ }
+ }
+
+ props = buildProperties(table);
Review comment:
ok, let me edit it.
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -339,15 +348,29 @@ public Builder setSnapshotProperty(String property, String value) {
*/
@Deprecated
public DataStreamSink<RowData> build() {
- return chainIcebergOperators();
+ return append();
}
/**
* Append the iceberg sink operators to write records to iceberg table.
*
* @return {@link DataStreamSink} for sink.
*/
- public DataStreamSink<Void> append() {
+ @SuppressWarnings("rawtypes")
Review comment:
Reference here:https://github.com/apache/iceberg/pull/3998#discussion_r812534842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r815385051
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -322,6 +322,19 @@ public Builder setSnapshotProperty(String property, String value) {
*/
@Deprecated
public DataStreamSink<RowData> build() {
Review comment:
There is a problem. Their return value types are inconsistent. Is there a solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r821346863
##########
File path: flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java
##########
@@ -326,4 +327,51 @@ public void testTwoSinksInDisjointedDAG() throws Exception {
Assert.assertEquals("rightTable", rightTable.currentSnapshot().summary().get("direction"));
}
+ @Test
+ public void testOverrideWriteConfigWithUnknownDistributionMode() {
+ AssertHelpers.assertThrows("Override write configuration.",
+ RuntimeException.class, "No enum constant org.apache.iceberg.DistributionMode.UNRECOGNIZED",
+ () -> {
+ Map<String, String> newProps = Maps.newHashMap();
+ newProps.put(TableProperties.WRITE_DISTRIBUTION_MODE, "UNRECOGNIZED");
+
+ List<Row> rows = createRows("");
+ DataStream<Row> dataStream = env.addSource(createBoundedSource(rows), ROW_TYPE_INFO);
+
+ FlinkSink.forRow(dataStream, SimpleDataUtil.FLINK_SCHEMA)
+ .table(table)
+ .tableLoader(tableLoader)
+ .writeParallelism(parallelism)
+ .properties(newProps)
+ .append();
+
+ // Execute the program.
+ env.execute("Test Iceberg DataStream.");
Review comment:
I've taken out the code that doesn't need to be put in the assertion. It does look more rigorous and elegant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r812536482
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -375,7 +387,7 @@ private String operatorName(String suffix) {
}
// Fallback to use upsert mode parsed from table properties if don't specify in job level.
- boolean upsertMode = upsert || PropertyUtil.propertyAsBoolean(table.properties(),
+ boolean upsertMode = upsert || PropertyUtil.propertyAsBoolean(props,
Review comment:
I think Ryan meant passing Properties like `distributeDataStream` method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r810467766
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -331,6 +342,7 @@ public Builder setSnapshotProperty(String property, String value) {
* @return {@link DataStreamSink} for sink.
*/
public DataStreamSink<Void> append() {
+ buildProperties();
Review comment:
I've done the first two steps here, building the table first and merging the configuration. In the third step, instead of passing table and props into the chainIcebergOperators, I directly use member field in chainIcebergOperators, is this feasible? thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on pull request #3998: Flink: support overwrite configuration in the flink sink action
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#issuecomment-1027610771
initially, I misread the intention was to do partition overwrite for commit
```
public Builder overwrite(boolean newOverwrite) {
this.overwrite = newOverwrite;
return this;
}
```
Can we update the PR title to sth like "Support override write configuration on top of table properties in FlinkSink builder"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r819774784
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -515,14 +549,21 @@ static RowType toFlinkRowType(Schema schema, TableSchema requestedSchema) {
RowType flinkRowType,
List<Integer> equalityFieldIds,
boolean upsert) {
+ return createStreamWriter(table, table.properties(), flinkRowType, equalityFieldIds, upsert);
Review comment:
Is this still called? Why preserve this method if it isn't public?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r816660089
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -375,7 +387,7 @@ private String operatorName(String suffix) {
}
// Fallback to use upsert mode parsed from table properties if don't specify in job level.
- boolean upsertMode = upsert || PropertyUtil.propertyAsBoolean(table.properties(),
+ boolean upsertMode = upsert || PropertyUtil.propertyAsBoolean(props,
Review comment:
ok, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r821295642
##########
File path: flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java
##########
@@ -326,4 +327,51 @@ public void testTwoSinksInDisjointedDAG() throws Exception {
Assert.assertEquals("rightTable", rightTable.currentSnapshot().summary().get("direction"));
}
+ @Test
+ public void testOverrideWriteConfigWithUnknownDistributionMode() {
+ AssertHelpers.assertThrows("Override write configuration.",
Review comment:
Learned, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r815384584
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -322,6 +322,19 @@ public Builder setSnapshotProperty(String property, String value) {
*/
@Deprecated
public DataStreamSink<RowData> build() {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r837007918
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -339,15 +348,29 @@ public Builder setSnapshotProperty(String property, String value) {
*/
@Deprecated
public DataStreamSink<RowData> build() {
- return chainIcebergOperators();
+ return append();
}
/**
* Append the iceberg sink operators to write records to iceberg table.
*
* @return {@link DataStreamSink} for sink.
*/
- public DataStreamSink<Void> append() {
+ @SuppressWarnings("rawtypes")
Review comment:
Why change this public method's return data type ? Be careful it's a incompatibility changes. In the first version, w e introduced the `FlinkSink#build` method with an incorrect return data type, and then we marked it as deprecated in this [PR](https://github.com/apache/iceberg/pull/3061)
If any changes are planning to make in this PR, then we will need to follow the same approach. In my thought, I don't think we need to change this, the current `DataStreamSink<Void>` is the correct return type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r804827121
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -168,6 +170,11 @@ public Builder table(Table newTable) {
return this;
}
+ public Builder properties(Map<String, String> newProps) {
Review comment:
May I suggest the following? I don't think we need to add trivial `@param` and `@return`
```
/**
* Set the write properties for Flink sink. Write properties (with the same keys) defined in table properties
* will be overwritten by properties provided here.
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r805089888
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -168,6 +170,11 @@ public Builder table(Table newTable) {
return this;
}
+ public Builder properties(Map<String, String> newProps) {
Review comment:
thx. done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3998: Flink: support overwrite configuration in the flink sink action
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r797292534
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -430,6 +440,19 @@ private String operatorName(String suffix) {
throw new RuntimeException("Unrecognized write.distribution-mode: " + writeMode);
}
}
+
+ private void buildProperties() {
+ if (table == null) {
+ return;
+ }
+ if (this.props.isEmpty()) {
+ this.props = table.properties();
+ } else {
+ Map<String, String> result = Maps.newHashMap(this.table.properties());
+ this.props.forEach((key, value) -> result.merge(key, value, (v1, v2) -> v2));
Review comment:
is it simpler to just do `result.put(key, value)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r812535418
##########
File path: flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java
##########
@@ -326,4 +327,48 @@ public void testTwoSinksInDisjointedDAG() throws Exception {
Assert.assertEquals("rightTable", rightTable.currentSnapshot().summary().get("direction"));
}
+ @Test
+ public void testOverrideWriteConfiguration() {
Review comment:
nit: make the test method name more descriptive. like `testOverrideWriteConfigWithUnknownDistributionMode`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#issuecomment-1066576528
cc @rdblue , PTAL, thx.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3998: Flink: support overwrite configuration in the flink sink action
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r796203579
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -288,7 +295,7 @@ public Builder uidPrefix(String newPrefix) {
// Distribute the records from input data stream based on the write.distribution-mode.
DataStream<RowData> distributeStream = distributeDataStream(
- rowDataInput, table.properties(), table.spec(), table.schema(), flinkRowType);
+ rowDataInput, this.props, table.spec(), table.schema(), flinkRowType);
Review comment:
Nit: We don't use `this` to access instance variables typically other than in the constructors and places it's absolutely needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r810204143
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -375,7 +387,7 @@ private String operatorName(String suffix) {
}
// Fallback to use upsert mode parsed from table properties if don't specify in job level.
- boolean upsertMode = upsert || PropertyUtil.propertyAsBoolean(table.properties(),
+ boolean upsertMode = upsert || PropertyUtil.propertyAsBoolean(props,
Review comment:
Properties should be passed in here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r812534842
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -322,6 +322,19 @@ public Builder setSnapshotProperty(String property, String value) {
*/
@Deprecated
public DataStreamSink<RowData> build() {
Review comment:
`build` can forward the execution to `append`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r815384578
##########
File path: flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java
##########
@@ -326,4 +327,48 @@ public void testTwoSinksInDisjointedDAG() throws Exception {
Assert.assertEquals("rightTable", rightTable.currentSnapshot().summary().get("direction"));
}
+ @Test
+ public void testOverrideWriteConfiguration() {
Review comment:
good.
##########
File path: flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java
##########
@@ -326,4 +327,48 @@ public void testTwoSinksInDisjointedDAG() throws Exception {
Assert.assertEquals("rightTable", rightTable.currentSnapshot().summary().get("direction"));
}
+ @Test
+ public void testOverrideWriteConfiguration() {
+ AssertHelpers.assertThrows("Override write configuration.",
+ RuntimeException.class, "No enum constant org.apache.iceberg.DistributionMode.UNRECOGNIZED",
+ () -> {
+ Map<String, String> newProps = Maps.newHashMap();
+ newProps.put(TableProperties.WRITE_DISTRIBUTION_MODE, "UNRECOGNIZED");
+
+ List<Row> rows = createRows("");
+ DataStream<Row> dataStream = env.addSource(createBoundedSource(rows), ROW_TYPE_INFO);
+
+ FlinkSink.forRow(dataStream, SimpleDataUtil.FLINK_SCHEMA)
+ .table(table)
+ .tableLoader(tableLoader)
+ .writeParallelism(parallelism)
+ .properties(newProps)
+ .append();
+
+ // Execute the program.
+ env.execute("Test Iceberg DataStream.");
+ return null;
+ });
+
+ AssertHelpers.assertThrows("Override write configuration.",
Review comment:
done
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -322,6 +322,19 @@ public Builder setSnapshotProperty(String property, String value) {
*/
@Deprecated
public DataStreamSink<RowData> build() {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r819770324
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -322,6 +322,19 @@ public Builder setSnapshotProperty(String property, String value) {
*/
@Deprecated
public DataStreamSink<RowData> build() {
Review comment:
Since this returns the same actual object, it should be fine. How about just adding an unchecked cast with a suppression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r819775820
##########
File path: flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java
##########
@@ -326,4 +327,51 @@ public void testTwoSinksInDisjointedDAG() throws Exception {
Assert.assertEquals("rightTable", rightTable.currentSnapshot().summary().get("direction"));
}
+ @Test
+ public void testOverrideWriteConfigWithUnknownDistributionMode() {
+ AssertHelpers.assertThrows("Override write configuration.",
Review comment:
This context should explain what you're testing a bit better, like "Should fail with invalid distribution mode"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r821295893
##########
File path: flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java
##########
@@ -326,4 +327,51 @@ public void testTwoSinksInDisjointedDAG() throws Exception {
Assert.assertEquals("rightTable", rightTable.currentSnapshot().summary().get("direction"));
}
+ @Test
+ public void testOverrideWriteConfigWithUnknownDistributionMode() {
+ AssertHelpers.assertThrows("Override write configuration.",
+ RuntimeException.class, "No enum constant org.apache.iceberg.DistributionMode.UNRECOGNIZED",
Review comment:
Replaced with IllegalArgumentException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#issuecomment-1081530559
> I'm okay to add customized properties to override the table properties for a given write flink job, BUT I'm very concern that the current PR did not make the merged properties go though the whole write path because I still see lots of places in write path just use the table properties to parse the configured keys ( That means the customized properties will be useless , although we've customized it).
>
> For example, The [RowDataTaskWriterFactory](https://github.com/apache/iceberg/blob/340a0c5f30a7fb1e33dde4c1f082fd4cb9f7472f/flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java#L88) will use the serialized table to initialize the `OutputFileFactory` which will use the table properties ....
Thanks for the correction. Let me go through as many end to end as I can and see what I'm missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r810468140
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -375,7 +387,7 @@ private String operatorName(String suffix) {
}
// Fallback to use upsert mode parsed from table properties if don't specify in job level.
- boolean upsertMode = upsert || PropertyUtil.propertyAsBoolean(table.properties(),
+ boolean upsertMode = upsert || PropertyUtil.propertyAsBoolean(props,
Review comment:
Are you saying that I should pass the props instead of setting it as a member variable as follow?
https://github.com/apache/iceberg/blob/2ca57336533e1a8a15184006bee77994ca4bf161/flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java#L136
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r819772923
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -174,6 +175,15 @@ public Builder table(Table newTable) {
return this;
}
+ /**
+ * Set the write properties for Flink sink. Write properties (with the same keys) defined in table properties
+ * will be overwritten by properties provided here.
+ */
+ public Builder properties(Map<String, String> newProps) {
Review comment:
Other APIs use `set` for individual key/value properties and `setAll` for a map of properties. Can you update this to `set` or `setProperties`? It isn't clear what `properties` does.
In the `Parquet` write builder, for example, we have these methods:
```java
public WriteBuilder set(String property, String value) {
config.put(property, value);
return this;
}
public WriteBuilder setAll(Map<String, String> properties) {
config.putAll(properties);
return this;
}
```
This should also add properties, not replace previous properties set by this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r819772923
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -174,6 +175,15 @@ public Builder table(Table newTable) {
return this;
}
+ /**
+ * Set the write properties for Flink sink. Write properties (with the same keys) defined in table properties
+ * will be overwritten by properties provided here.
+ */
+ public Builder properties(Map<String, String> newProps) {
Review comment:
Other APIs use `set` for individual key/value properties and `setAll` for a map of properties. Can you update this to `set` or `setProperties`? It isn't clear what `properties` does.
In the `Parquet` write builder, for example, we have these methods:
```java
public WriteBuilder set(String property, String value) {
config.put(property, value);
return this;
}
public WriteBuilder setAll(Map<String, String> properties) {
config.putAll(properties);
return this;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r819777087
##########
File path: flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java
##########
@@ -326,4 +327,51 @@ public void testTwoSinksInDisjointedDAG() throws Exception {
Assert.assertEquals("rightTable", rightTable.currentSnapshot().summary().get("direction"));
}
+ @Test
+ public void testOverrideWriteConfigWithUnknownDistributionMode() {
+ AssertHelpers.assertThrows("Override write configuration.",
+ RuntimeException.class, "No enum constant org.apache.iceberg.DistributionMode.UNRECOGNIZED",
Review comment:
RuntimeException is too broad. Can you test a more specific exception? Otherwise anything that goes wrong in this huge block looks like the test is passing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r837008330
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -339,15 +348,29 @@ public Builder setSnapshotProperty(String property, String value) {
*/
@Deprecated
public DataStreamSink<RowData> build() {
- return chainIcebergOperators();
+ return append();
}
/**
* Append the iceberg sink operators to write records to iceberg table.
*
* @return {@link DataStreamSink} for sink.
*/
- public DataStreamSink<Void> append() {
+ @SuppressWarnings("rawtypes")
+ public DataStreamSink append() {
+ Preconditions.checkNotNull(tableLoader, "Table loader shouldn't be null");
+
+ if (table == null) {
+ tableLoader.open();
+ try (TableLoader loader = tableLoader) {
+ this.table = loader.loadTable();
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to load iceberg table from table loader: " + tableLoader, e);
+ }
+ }
+
+ props = buildProperties(table);
Review comment:
It seems to be unnecessary to define this `buildProperties` method, how about just moving its body into this `append` method ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r810203572
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -442,6 +455,16 @@ private String operatorName(String suffix) {
throw new RuntimeException("Unrecognized write.distribution-mode: " + writeMode);
}
}
+
+ private void buildProperties() {
Review comment:
I think this should accept a Table instance and return a merged properties map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r810203962
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -331,6 +342,7 @@ public Builder setSnapshotProperty(String property, String value) {
* @return {@link DataStreamSink} for sink.
*/
public DataStreamSink<Void> append() {
+ buildProperties();
Review comment:
This needs to build the table first, then pass the table in to build the properties map. Then it should pass the table and the properties into chainIcebergOperators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r796203579
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -288,7 +295,7 @@ public Builder uidPrefix(String newPrefix) {
// Distribute the records from input data stream based on the write.distribution-mode.
DataStream<RowData> distributeStream = distributeDataStream(
- rowDataInput, table.properties(), table.spec(), table.schema(), flinkRowType);
+ rowDataInput, this.props, table.spec(), table.schema(), flinkRowType);
Review comment:
Nit: We don't use `this` to access instance variables typically other than in the constructors and places that we modify class properties directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r812535255
##########
File path: flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java
##########
@@ -326,4 +327,48 @@ public void testTwoSinksInDisjointedDAG() throws Exception {
Assert.assertEquals("rightTable", rightTable.currentSnapshot().summary().get("direction"));
}
+ @Test
+ public void testOverrideWriteConfiguration() {
+ AssertHelpers.assertThrows("Override write configuration.",
+ RuntimeException.class, "No enum constant org.apache.iceberg.DistributionMode.UNRECOGNIZED",
+ () -> {
+ Map<String, String> newProps = Maps.newHashMap();
+ newProps.put(TableProperties.WRITE_DISTRIBUTION_MODE, "UNRECOGNIZED");
+
+ List<Row> rows = createRows("");
+ DataStream<Row> dataStream = env.addSource(createBoundedSource(rows), ROW_TYPE_INFO);
+
+ FlinkSink.forRow(dataStream, SimpleDataUtil.FLINK_SCHEMA)
+ .table(table)
+ .tableLoader(tableLoader)
+ .writeParallelism(parallelism)
+ .properties(newProps)
+ .append();
+
+ // Execute the program.
+ env.execute("Test Iceberg DataStream.");
+ return null;
+ });
+
+ AssertHelpers.assertThrows("Override write configuration.",
Review comment:
nit: can separate it to a test method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r819777708
##########
File path: flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java
##########
@@ -326,4 +327,51 @@ public void testTwoSinksInDisjointedDAG() throws Exception {
Assert.assertEquals("rightTable", rightTable.currentSnapshot().summary().get("direction"));
}
+ @Test
+ public void testOverrideWriteConfigWithUnknownDistributionMode() {
+ AssertHelpers.assertThrows("Override write configuration.",
+ RuntimeException.class, "No enum constant org.apache.iceberg.DistributionMode.UNRECOGNIZED",
+ () -> {
+ Map<String, String> newProps = Maps.newHashMap();
+ newProps.put(TableProperties.WRITE_DISTRIBUTION_MODE, "UNRECOGNIZED");
+
+ List<Row> rows = createRows("");
+ DataStream<Row> dataStream = env.addSource(createBoundedSource(rows), ROW_TYPE_INFO);
+
+ FlinkSink.forRow(dataStream, SimpleDataUtil.FLINK_SCHEMA)
+ .table(table)
+ .tableLoader(tableLoader)
+ .writeParallelism(parallelism)
+ .properties(newProps)
+ .append();
+
+ // Execute the program.
+ env.execute("Test Iceberg DataStream.");
+ return null;
+ });
+ }
+
+ @Test
+ public void testOverrideWriteConfigWithUnknownFileFormat() {
Review comment:
Mostly the same comments for this test as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r819775347
##########
File path: flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java
##########
@@ -326,4 +327,51 @@ public void testTwoSinksInDisjointedDAG() throws Exception {
Assert.assertEquals("rightTable", rightTable.currentSnapshot().summary().get("direction"));
}
+ @Test
+ public void testOverrideWriteConfigWithUnknownDistributionMode() {
+ AssertHelpers.assertThrows("Override write configuration.",
+ RuntimeException.class, "No enum constant org.apache.iceberg.DistributionMode.UNRECOGNIZED",
Review comment:
Indentation in this class is off. Can you use 4 spaces for continuation indents?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on a change in pull request #3998: Flink: Support override write configuration on top of table properties in FlinkSink builder
Posted by GitBox <gi...@apache.org>.
hililiwei commented on a change in pull request #3998:
URL: https://github.com/apache/iceberg/pull/3998#discussion_r805090103
##########
File path: flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -288,7 +295,7 @@ public Builder uidPrefix(String newPrefix) {
// Distribute the records from input data stream based on the write.distribution-mode.
DataStream<RowData> distributeStream = distributeDataStream(
- rowDataInput, table.properties(), table.spec(), table.schema(), flinkRowType);
+ rowDataInput, this.props, table.spec(), table.schema(), flinkRowType);
Review comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org