You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "tomerr90 (via GitHub)" <gi...@apache.org> on 2026/04/06 20:08:02 UTC

[I] Flink: In upsert mode with equality fields on a partitioned table, writeParallelism is ignored for validations [iceberg]

tomerr90 opened a new issue, #15900:
URL: https://github.com/apache/iceberg/issues/15900

   ### Apache Iceberg version
   
   1.10.1 (latest release)
   
   ### Query engine
   
   Flink
   
   ### Please describe the bug 🐞
   
   1. https://github.com/apache/iceberg/blob/main/flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java#L952
   
   2. https://github.com/apache/iceberg/blob/main/flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java#L755
   
   Both of the above are only relevant for writeParallelism > 1, otherwise everything goes to the same sink / subtask and everything should work fine
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Flink: In upsert mode with equality fields on a partitioned table, partition column has to be in equalityFields [iceberg]

Posted by "tomerr90 (via GitHub)" <gi...@apache.org>.
tomerr90 commented on issue #15900:
URL: https://github.com/apache/iceberg/issues/15900#issuecomment-4297542318

   Apologies, just thought about it again and realised Im wrong, the partition key needs to be consistent and not changing and create_time is like that so I can add it as an equality field, even though its a bit inefficient, but it will still work


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Flink: In upsert mode with equality fields on a partitioned table, partition column has to be in equalityFields [iceberg]

Posted by "tomerr90 (via GitHub)" <gi...@apache.org>.
tomerr90 closed issue #15900: Flink: In upsert mode with equality fields on a partitioned table, partition column has to be in equalityFields
URL: https://github.com/apache/iceberg/issues/15900


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Flink: In upsert mode with equality fields on a partitioned table, partition column has to be in equalityFields [iceberg]

Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on issue #15900:
URL: https://github.com/apache/iceberg/issues/15900#issuecomment-4195806676

   Hi @tomer90 - thanks for raising this issue.
   
   I took a look at the second code reference you shared and I suspect one of the two are true about the invariant for partition source IDs to be a part of equality fields for upserts:
   1. The invariant is only needed for **hash distribution mode**. Then the implementation is wrong and should gate this check on the actual distribution mode like you suggest.
   2. The invariant is correct **independent of distribution mode**. Then the error message is simply wrong and should be updated.
   
   We may benefit from pulling in Flink Iceberg experts here, but based on the [Javadocs](https://iceberg.apache.org/javadoc/1.9.2/org/apache/iceberg/flink/sink/FlinkSink.Builder.html#upsert(boolean)) I think the second conclusion is likely to be true.
   
   Let me know if the explanation in the javadocs aligns with your understanding of how upserts should work as well


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Flink: In upsert mode with equality fields on a partitioned table, partition column has to be in equalityFields [iceberg]

Posted by "tomerr90 (via GitHub)" <gi...@apache.org>.
tomerr90 commented on issue #15900:
URL: https://github.com/apache/iceberg/issues/15900#issuecomment-4292572307

   Thanks Sung!
   I feel like I have to be missing something here, an Iceberg table partitioned by a function over a timestamp field is probably the most common use case of Iceberg, so I find it very weird that this common case basically doesnt work in Flink upsert because it forces you to use the timestamp field (without the function even, just the field) as an equality field for the delete.
   
   @mxm @rodmeneses @Guosmilesmile sorry for the tag, PLMK if thats not OK, I saw you guys worked in the IcebergSink recently, would greatly appreciate your take on the above, seems like this "double" use of equalityFields for deletion equality and also flink stream partitioning is limiting common use cases, maybe the `keyBy` in the sink should transparently add the partitioning functions of the table AND the equalityFields? So equality is still only checked by the equalityFields but we maintain correctness of the deletes.
   Also, in general, this whole discussion is only relevant if writeParallelism is > 1, agree?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Flink: In upsert mode with equality fields on a partitioned table, partition column has to be in equalityFields [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on issue #15900:
URL: https://github.com/apache/iceberg/issues/15900#issuecomment-4305077586

   Just for context: We have to make sure a given equality key value gets processed by the only one writer. Otherwise, multiple writers would be writing delete files which could conflict. For non-partitioned table, we partition by the equality value itself. For partitioned table, like in your case, we enforce that the equality values are a subset of the partition key.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org