You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Aitozi (via GitHub)" <gi...@apache.org> on 2026/05/12 13:32:57 UTC

[PR] [core] Support writing empty partitions [paimon]

Aitozi opened a new pull request, #7830:
URL: https://github.com/apache/paimon/pull/7830

   ### Purpose
   
   This change adds support for writing an empty data file for static partition overwrite when the input produces no records and `write.empty.partition.enable` is enabled.
   
   ### Tests
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Support writing empty partitions [paimon]

Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on PR #7830:
URL: https://github.com/apache/paimon/pull/7830#issuecomment-4587139019

   > It seems that the requirement is only to notify CommitListener of the partition information when Overwriting? I feel there is a simpler way to modify it.
   
   I have updated the implementation:
   
   1. Added a Writer interface to support writing EmptyFiles.
   2. In the Flink and Spark commit phases, empty partitions are now committed directly during static overwrite.
   
   Could you please help review it again? @JingsongLi 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Support writing empty partitions [paimon]

Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on PR #7830:
URL: https://github.com/apache/paimon/pull/7830#issuecomment-4748901995

   > Thanks @Aitozi for the update. But I still think something like `_SUCCESS` file helps this scenario. Empty file is too dangerous to Paimon.
   
   Thanks for your suggestion, closing it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Support writing empty partitions [paimon]

Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi closed pull request #7830: [core] Support writing empty partitions
URL: https://github.com/apache/paimon/pull/7830


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Support writing empty partitions [paimon]

Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on PR #7830:
URL: https://github.com/apache/paimon/pull/7830#issuecomment-4483647826

   CC @JingsongLi @Stefanietry 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Support writing empty partitions [paimon]

Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on PR #7830:
URL: https://github.com/apache/paimon/pull/7830#issuecomment-4499939507

   > This is a bit too tricky and has affected too much code. We should think of a simpler method.
   
   @JingsongLi Do you have some good suggestions for this? 
   I feel that expressing this type of empty partition within Paimon's existing architecture is a bit difficult. I think the biggest issue here is that Paimon doesn't have independent partition metadata, so it can only represent its partitions through files. Therefore, to indicate whether a partition exists, we need to represent it by creating empty files.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Support writing empty partitions [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #7830:
URL: https://github.com/apache/paimon/pull/7830#issuecomment-4660894179

   Thanks @Aitozi for the update. But I still think something like `_SUCCESS` file helps this scenario. Empty file is too dangerous to Paimon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org