You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Patrick Wendell (JIRA)" <ji...@apache.org> on 2012/11/06 17:46:12 UTC
[jira] [Created] (FLUME-1683) Fix Time Granularity Bug in
SpoolingFileLineReader
Patrick Wendell created FLUME-1683:
--------------------------------------
Summary: Fix Time Granularity Bug in SpoolingFileLineReader
Key: FLUME-1683
URL: https://issues.apache.org/jira/browse/FLUME-1683
Project: Flume
Issue Type: Bug
Reporter: Patrick Wendell
Assignee: Patrick Wendell
This patch fixes a test in the SpoolingFileLineReader, and a bug associated with that test:
The issue is related to the way that the source sorts files which are candidates for ingestion. It sorts the files by timestamp, and the unit test assumes that if it creates two files:
x = new File()
Files.write("something", x)
y = new File()
Files.write("something", y)
// Read from files
that x will be processed before y. In my filesystem, the time granularity is only 1 second, so these files were actually processed in a non-deterministic order. On your filesystem, they were processed in the correct order, and the test (correctly) failed.
I didn't notice that the test was mixed up because it was passing for me due to this issue.
I updated the sorting order to take into account filename when there is a tie in the timestamp, so at least we have a consistent ordering we can test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1683) Fix Time Granularity Bug in
SpoolingFileLineReader
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated FLUME-1683:
-----------------------------------
Attachment: FLUME-1683.v1.txt
> Fix Time Granularity Bug in SpoolingFileLineReader
> --------------------------------------------------
>
> Key: FLUME-1683
> URL: https://issues.apache.org/jira/browse/FLUME-1683
> Project: Flume
> Issue Type: Bug
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: FLUME-1683.v1.txt
>
>
> This patch fixes a test in the SpoolingFileLineReader, and a bug associated with that test:
> The issue is related to the way that the source sorts files which are candidates for ingestion. It sorts the files by timestamp, and the unit test assumes that if it creates two files:
> x = new File()
> Files.write("something", x)
> y = new File()
> Files.write("something", y)
> // Read from files
> that x will be processed before y. In my filesystem, the time granularity is only 1 second, so these files were actually processed in a non-deterministic order. On your filesystem, they were processed in the correct order, and the test (correctly) failed.
> I didn't notice that the test was mixed up because it was passing for me due to this issue.
> I updated the sorting order to take into account filename when there is a tie in the timestamp, so at least we have a consistent ordering we can test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1683) Fix Time Granularity Bug in
SpoolingFileLineReader
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated FLUME-1683:
-----------------------------------
Attachment: FLUME-1683.v2.txt
Small one line change removing extra exception.
> Fix Time Granularity Bug in SpoolingFileLineReader
> --------------------------------------------------
>
> Key: FLUME-1683
> URL: https://issues.apache.org/jira/browse/FLUME-1683
> Project: Flume
> Issue Type: Bug
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: FLUME-1683.v1.txt, FLUME-1683.v2.txt
>
>
> This patch fixes a test in the SpoolingFileLineReader, and a bug associated with that test:
> The issue is related to the way that the source sorts files which are candidates for ingestion. It sorts the files by timestamp, and the unit test assumes that if it creates two files:
> x = new File()
> Files.write("something", x)
> y = new File()
> Files.write("something", y)
> // Read from files
> that x will be processed before y. In my filesystem, the time granularity is only 1 second, so these files were actually processed in a non-deterministic order. On your filesystem, they were processed in the correct order, and the test (correctly) failed.
> I didn't notice that the test was mixed up because it was passing for me due to this issue.
> I updated the sorting order to take into account filename when there is a tie in the timestamp, so at least we have a consistent ordering we can test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1683) Fix Time Granularity Bug in
SpoolingFileLineReader
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492199#comment-13492199 ]
Hudson commented on FLUME-1683:
-------------------------------
Integrated in flume-trunk #322 (See [https://builds.apache.org/job/flume-trunk/322/])
FLUME-1683: Fix Time Granularity Bug in SpoolingFileLineReader (Revision c145b2b8c1ed7bc04e1b018085c519555cf61a21)
Result = ABORTED
brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=c145b2b8c1ed7bc04e1b018085c519555cf61a21
Files :
* flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java
* flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java
> Fix Time Granularity Bug in SpoolingFileLineReader
> --------------------------------------------------
>
> Key: FLUME-1683
> URL: https://issues.apache.org/jira/browse/FLUME-1683
> Project: Flume
> Issue Type: Bug
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: v1.3.0
>
> Attachments: FLUME-1683.v1.txt, FLUME-1683.v2.txt
>
>
> This patch fixes a test in the SpoolingFileLineReader, and a bug associated with that test:
> The issue is related to the way that the source sorts files which are candidates for ingestion. It sorts the files by timestamp, and the unit test assumes that if it creates two files:
> x = new File()
> Files.write("something", x)
> y = new File()
> Files.write("something", y)
> // Read from files
> that x will be processed before y. In my filesystem, the time granularity is only 1 second, so these files were actually processed in a non-deterministic order. On your filesystem, they were processed in the correct order, and the test (correctly) failed.
> I didn't notice that the test was mixed up because it was passing for me due to this issue.
> I updated the sorting order to take into account filename when there is a tie in the timestamp, so at least we have a consistent ordering we can test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1683) Fix Time Granularity Bug in
SpoolingFileLineReader
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491616#comment-13491616 ]
Brock Noland commented on FLUME-1683:
-------------------------------------
OK thanks! I am running unit tests now.
> Fix Time Granularity Bug in SpoolingFileLineReader
> --------------------------------------------------
>
> Key: FLUME-1683
> URL: https://issues.apache.org/jira/browse/FLUME-1683
> Project: Flume
> Issue Type: Bug
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: v1.3.0
>
> Attachments: FLUME-1683.v1.txt, FLUME-1683.v2.txt
>
>
> This patch fixes a test in the SpoolingFileLineReader, and a bug associated with that test:
> The issue is related to the way that the source sorts files which are candidates for ingestion. It sorts the files by timestamp, and the unit test assumes that if it creates two files:
> x = new File()
> Files.write("something", x)
> y = new File()
> Files.write("something", y)
> // Read from files
> that x will be processed before y. In my filesystem, the time granularity is only 1 second, so these files were actually processed in a non-deterministic order. On your filesystem, they were processed in the correct order, and the test (correctly) failed.
> I didn't notice that the test was mixed up because it was passing for me due to this issue.
> I updated the sorting order to take into account filename when there is a tie in the timestamp, so at least we have a consistent ordering we can test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (FLUME-1683) Fix Time Granularity Bug in
SpoolingFileLineReader
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland resolved FLUME-1683.
---------------------------------
Resolution: Fixed
Thank you Patrick! Committed to trunk, 1.3, and 1.4!
> Fix Time Granularity Bug in SpoolingFileLineReader
> --------------------------------------------------
>
> Key: FLUME-1683
> URL: https://issues.apache.org/jira/browse/FLUME-1683
> Project: Flume
> Issue Type: Bug
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: v1.3.0
>
> Attachments: FLUME-1683.v1.txt, FLUME-1683.v2.txt
>
>
> This patch fixes a test in the SpoolingFileLineReader, and a bug associated with that test:
> The issue is related to the way that the source sorts files which are candidates for ingestion. It sorts the files by timestamp, and the unit test assumes that if it creates two files:
> x = new File()
> Files.write("something", x)
> y = new File()
> Files.write("something", y)
> // Read from files
> that x will be processed before y. In my filesystem, the time granularity is only 1 second, so these files were actually processed in a non-deterministic order. On your filesystem, they were processed in the correct order, and the test (correctly) failed.
> I didn't notice that the test was mixed up because it was passing for me due to this issue.
> I updated the sorting order to take into account filename when there is a tie in the timestamp, so at least we have a consistent ordering we can test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1683) Fix Time Granularity Bug in
SpoolingFileLineReader
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated FLUME-1683:
--------------------------------
Fix Version/s: v1.3.0
> Fix Time Granularity Bug in SpoolingFileLineReader
> --------------------------------------------------
>
> Key: FLUME-1683
> URL: https://issues.apache.org/jira/browse/FLUME-1683
> Project: Flume
> Issue Type: Bug
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: v1.3.0
>
> Attachments: FLUME-1683.v1.txt, FLUME-1683.v2.txt
>
>
> This patch fixes a test in the SpoolingFileLineReader, and a bug associated with that test:
> The issue is related to the way that the source sorts files which are candidates for ingestion. It sorts the files by timestamp, and the unit test assumes that if it creates two files:
> x = new File()
> Files.write("something", x)
> y = new File()
> Files.write("something", y)
> // Read from files
> that x will be processed before y. In my filesystem, the time granularity is only 1 second, so these files were actually processed in a non-deterministic order. On your filesystem, they were processed in the correct order, and the test (correctly) failed.
> I didn't notice that the test was mixed up because it was passing for me due to this issue.
> I updated the sorting order to take into account filename when there is a tie in the timestamp, so at least we have a consistent ordering we can test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira