[jira] [Comment Edited] (NIFI-3332) Bug in ListXXX causes matching timestamps to be ignored on later runs

["Joe Skora (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15877439#comment-15877439 ] 

Joe Skora edited comment on NIFI-3332 at 2/22/17 4:15 AM:
----------------------------------------------------------

[~ijokarumawak] Admittedly, discussing time orientation and time references for these tests can get confusing, I realize now that t3 is older than t2 and so on, I had mixed it up earlier today.

Table 1 looks mostly right to me.  Since it is based on the old logic, the t3 run should output the batch1-age3.txt, batch1-age4.txt, and batch1-age5.txt files and store a state timestamp of t3 and processed files list containing just the batch1-age3.txt file since the batch1-age4.txt and batch1-age5.txt files are older than the state timestamp.  The subsequent t2 run looks correct to me, outputting the batch2-age3.txt and batch2-age2.txt files and storing state with the t2 timestamp and the batch2-age2.txt file.  So the Table 1 output information is correct except for the state file list on the t3 run containing files not matching the state timestamp.

I'm not sure I understand your "_SUCCESS" algorithm at this point so I can't comment on Table 2 until I get a chance to work through that tomorrow afternoon.


was (Author: jskora):
[~ijokarumawak] Admittedly, discussing time orientation and time references for these tests can get confusing, I think I understand now that t3 is older than t2 and so on.

Table 1 looks mostly right to me.  Since it is based on the old logic, the t3 run should output the batch1-age3.txt, batch1-age4.txt, and batch1-age5.txt files and store a state timestamp of t3 and processed files list containing just the batch1-age3.txt file since the batch1-age4.txt and batch1-age5.txt files are older than the state timestamp.  The subsequent t2 run looks correct to me, outputting the batch2-age3.txt and batch2-age2.txt files and storing state with the t2 timestamp and the batch2-age2.txt file.  So the Table 1 output information is correct except for the state file list on the t3 run containing files not matching the state timestamp.

I'm not sure I understand your "_SUCCESS" algorithm at this point so I can't comment on Table 2 until I get a chance to work through that tomorrow afternoon.

> Bug in ListXXX causes matching timestamps to be ignored on later runs
> ---------------------------------------------------------------------
>
>                 Key: NIFI-3332
>                 URL: https://issues.apache.org/jira/browse/NIFI-3332
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 0.7.1, 1.1.1
>            Reporter: Joe Skora
>            Assignee: Koji Kawamura
>            Priority: Critical
>         Attachments: Test-showing-ListFile-timestamp-bug.log, Test-showing-ListFile-timestamp-bug.patch
>
>
> The new state implementation for the ListXXX processors based on AbstractListProcessor creates a race conditions when processor runs occur while a batch of files is being written with the same timestamp.
> The changes to state management dropped tracking of the files processed for a given timestamp.  Without the record of files processed, the remainder of the batch is ignored on the next processor run since their timestamp is not greater than the one timestamp stored in processor state.  With the file tracking it was possible to process files that matched the timestamp exactly and exclude the previously processed files.
> A basic time goes as follows.
>   T0 - system creates or receives batch of files with Tx timestamp where Tx is more than the current timestamp in processor state.
>   T1 - system writes 1st half of Tx batch to the ListFile source directory.
>   T2 - ListFile runs picking up 1st half of Tx batch and stores Tx timestamp in processor state.
>   T3 - system writes 2nd half of Tx batch to ListFile source directory.
>   T4 - ListFile runs ignoring any files with T <= Tx, eliminating 2nd half Tx timestamp batch.
> I've attached a patch[1] for TestListFile.java that adds an instrumented unit test demonstrates the problem and a log[2] of the output from one such run.  The test writes 3 files each in two batches with processor runs after each batch.  Batch 2 writes files with timestamps older than, equal to, and newer than the timestamp stored when batch 1 was processed, but only the newer file is picked up.  The older file is correctly ignored but file with the matchin timestamp file should have been processed.
> [1] Test-showing-ListFile-timestamp-bug.patch
> [2] Test-showing-ListFile-timestamp-bug.log



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)