[jira] [Commented] (NIFI-8727) claimantCount will never decrement to zero

["Mark Bean (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17420942#comment-17420942 ] 

Mark Bean commented on NIFI-8727:
---------------------------------

I applied the same fix as above. Specifically, modified StandardProcessSession.clone() such that the new repository record included a reference to the current record:

final StandardRepositoryRecord record = new StandardRepositoryRecord(null, currRec);

However, this did not resolve the problem completely. Through additional log messages I added for debugging, I could see that the claimant count went to zero. However, in the flowfile repo (RocksDBFlowFileRepository.determineDestructibleClaims), I noted the "current claim" is destructible, but the "original claim" is not. Yet, when NiFi was restarted, that claim was identified as "Found unknown file" which means there were no flowfiles associated with this claim.


> claimantCount will never decrement to zero
> ------------------------------------------
>
>                 Key: NIFI-8727
>                 URL: https://issues.apache.org/jira/browse/NIFI-8727
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 1.12.0, 1.13.0, 1.12.1, 1.13.1, 1.13.2
>         Environment: linux
>            Reporter: wangliqiang
>            Priority: Major
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> When running my processor code below :
> {code:java}
> //originalFlowFile has content , so ClaimantCount=1
>  FlowFile multiFlowFile = session.clone(originalFlowFile); // claim count +1，so ClaimantCount=2
>  multiFlowFile = session.write(multiFlowFile, new OutputStreamCallback() {
>      @Override
>      public void process(OutputStream out) throws IOException {
>         IOUtils.write(tvMultiAlbumJson, out, Charset.forName("UTF-8"));
>      }
>  });//the new content will resuse the same resource claim , so ClaimantCount=3
>  //At this point we have two flowfile and two contentClaim ，and ClaimCount=3.
>  //When this two flowfiles dropped，the claimantCount should decrement to 0，however the result is ClaimantCount=1！
>  //If we use "sh nifi.sh diagnostics --verbose dump.log" to get a dump log，we will find some info like this “default/465/1623853427574-10705, Claimant Count = 1, In Use = true, Awaiting Destruction = false, References (0) = []” 
>  //And the file “default/465/1623853427574-10705” will never be archived，and will never be destroyed，and the content_repository will use more storage than it configs.{code}
> The above is a sort of phenomenon. The reason is the code below：
> {code:java}
> //session.clone
>  public FlowFile clone(FlowFile example, final long offset, final long size) {
>      .....................................
>      final StandardRepositoryRecord record = new StandardRepositoryRecord(null); //here the originalFlowFileRecord of record is null
>      .....................................
>      return clone;
>  }
>  //session.commit
>  private void updateClaimCounts(final RepositoryRecord record) {
>      ..........................................................
>      if (record.isContentModified()) {
>      decrementClaimCount(originalClaim); //here the originalClaim is null
>      }
>  }{code}
> Perhaps we should not use session.clone like that，but without official note it will sometimes happen to be used.
> So i change "final StandardRepositoryRecord record = new StandardRepositoryRecord(null)" to "final StandardRepositoryRecord record = new StandardRepositoryRecord(null, currRec);"
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)