You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by pvillard31 <gi...@git.apache.org> on 2016/06/02 23:13:18 UTC
[GitHub] nifi pull request #490: NIFI-1959 Added length and timestamp to detect rollo...
GitHub user pvillard31 opened a pull request:
https://github.com/apache/nifi/pull/490
NIFI-1959 Added length and timestamp to detect rollover
It improves robustness of the processor in case the property \u201cRolling filename pattern\u201d is not set and the tailed file is moved for some reasons.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pvillard31/nifi NIFI-1959
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/490.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #490
----
commit d3d59d58a51ded47ba1b7ebf87a79f678ab41cf2
Author: Pierre Villard <pi...@gmail.com>
Date: 2016-06-02T23:09:53Z
NIFI-1959 Added length and timestamp to detect rollover
It improves robustness of the processor in case the property \u201cRolling
filename pattern\u201d is not set and the tailed file is moved for some
reasons.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #490: NIFI-1959 Added length and timestamp to detect rollover
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/490
@pvillard31 I think you should be able to leverage the checksum in order to check if the file that is there is the one that was previously read. Maybe convert the logic found [here](https://github.com/apache/nifi/blob/d3d59d58a51ded47ba1b7ebf87a79f678ab41cf2/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java#L791-L791) into it's own function and reuse it. This might need to be a configurable property though (not sure about the performance implications, esp for an extreme corner case).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #490: NIFI-1959 Added length and timestamp to detect rollover
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/490
It remains an issue I have not been able to solve at this point: the case where the new file contains more data than the tailed file before being moved. In this case we ends up in the same situation: the new data in not ingested and the processor is not yielded. This situation should not happen really often... but still... if someone has an idea.
To reproduce: tail on /tmp/test.log
echo "test" >> /tmp/test.log
rm /tmp/test.log
echo "testtest" >> /tmp/test.log
In this case, both timestamp and size cannot be used to detect a pseudo-rollover and we are stuck with the previous state and reader. Not sure to see an easy solution.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #490: NIFI-1959 Added length and timestamp to detect rollover
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/490
Hey @pvillard31, I'm gonna start reviewing this. I noticed this was initially committed back in June, sorry no one has reviewed this yet but could you rebase? I know there are no conflicts but a lot has happened on the master branch in last month that could have an impact.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #490: NIFI-1959 Added length and timestamp to detect rollo...
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 closed the pull request at:
https://github.com/apache/nifi/pull/490
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #490: NIFI-1959 Added length and timestamp to detect rollover
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/490
Closing - implemented with #980
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---