You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "AkisAya (via GitHub)" <gi...@apache.org> on 2024/06/06 06:49:52 UTC

Re: [I] [Bug] The partition expire was not correctly triggered during the commit execution. [paimon]

AkisAya commented on issue #3434:
URL: https://github.com/apache/paimon/issues/3434#issuecomment-2151532695

   in a stream env, StoreCommitter holds TableCommitImpl, so expiration of partition works as expected。But in a batch env, if you create TableCommitImpl every time, sure there will be a new PartitionExpire every time and PartitionExpire#expire() will not be executed。but in both stream env and batch env, paimon uses TableCommitImpl to commit changes, there's no other way to differ it's a batch env or stream env. 
   ```
   class TableCommitImpl implements InnerTableCommit
   
   interface InnerTableCommit extends StreamTableCommit, BatchTableCommit
   ```
   
   i  think maybe we can add a commit method to distinguish diffrent env, like this
   ```
   
   
   public interface TableCommit extends AutoCloseable {
   
       /** Abort an unsuccessful commit. The data files will be deleted. */
       void abort(List<CommitMessage> commitMessages);
      /** newly add **/
      void commit (List<CommitMessage> commitMessages, EnvIdentifier env)
   }
   
   ```
   
   both StreamTableCommit and BatchTableCommit adapt to this new commit method.
   
   Overall, Paimon is a stream-first computing engine, and some designs do not take batch processing into consideration.
   or maybe they want you to use a seperate Partition Expiration to doExpire. just like what they do in PartitionExpireTest, to new a Expire to do Expiration.  And maybe there should be a PartitionExpirationProcedure for spark engine too.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org