You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Prasad Mujumdar (Created) (JIRA)" <ji...@apache.org> on 2012/04/06 02:40:23 UTC
[jira] [Created] (FLUME-1104) HDFS file roller creates causes the
first file to roll too soon
HDFS file roller creates causes the first file to roll too soon
---------------------------------------------------------------
Key: FLUME-1104
URL: https://issues.apache.org/jira/browse/FLUME-1104
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v1.2.0
Reporter: Prasad Mujumdar
Assignee: Prasad Mujumdar
The sink process() keep tracks of the buckets opened during the transaction. At the end of transaction, we need to flush all the buckets that has pending data. This is required in order to ensure that the data removed from channel should be safely in HDFS during commit.
Currently the files are tracked only when they are created and also getting closed during the cleanup instead of flush.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (FLUME-1104) HDFS rolls the first file
incorrectly
Posted by "Arvind Prabhakar (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arvind Prabhakar resolved FLUME-1104.
-------------------------------------
Resolution: Fixed
Fix Version/s: v1.2.0
Patch committed. Thanks Prasad!
> HDFS rolls the first file incorrectly
> -------------------------------------
>
> Key: FLUME-1104
> URL: https://issues.apache.org/jira/browse/FLUME-1104
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Fix For: v1.2.0
>
> Attachments: FLUME-1104-2.patch
>
>
> The sink process() keep tracks of the buckets opened during the transaction. At the end of transaction, we need to flush all the buckets that has pending data. This is required in order to ensure that the data removed from channel should be safely in HDFS during commit.
> Currently the files are tracked only when they are created and also getting closed during the cleanup instead of flush.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1104) HDFS rolls the first file
incorrectly
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248036#comment-13248036 ]
Hudson commented on FLUME-1104:
-------------------------------
Integrated in flume-trunk #161 (See [https://builds.apache.org/job/flume-trunk/161/])
FLUME-1104. HDFS Sink rolls the first file incorrectly.
(Prasad Mujumdar via Arvind Prabhakar) (Revision 1310160)
Result = SUCCESS
arvind : http://svn.apache.org/viewvc/?view=rev&rev=1310160
Files :
* /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
* /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
* /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
> HDFS rolls the first file incorrectly
> -------------------------------------
>
> Key: FLUME-1104
> URL: https://issues.apache.org/jira/browse/FLUME-1104
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Fix For: v1.2.0
>
> Attachments: FLUME-1104-2.patch
>
>
> The sink process() keep tracks of the buckets opened during the transaction. At the end of transaction, we need to flush all the buckets that has pending data. This is required in order to ensure that the data removed from channel should be safely in HDFS during commit.
> Currently the files are tracked only when they are created and also getting closed during the cleanup instead of flush.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1104) HDFS rolls the first file incorrectly
Posted by "Prasad Mujumdar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar updated FLUME-1104:
-----------------------------------
Attachment: FLUME-1104-2.patch
> HDFS rolls the first file incorrectly
> -------------------------------------
>
> Key: FLUME-1104
> URL: https://issues.apache.org/jira/browse/FLUME-1104
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: FLUME-1104-2.patch
>
>
> The sink process() keep tracks of the buckets opened during the transaction. At the end of transaction, we need to flush all the buckets that has pending data. This is required in order to ensure that the data removed from channel should be safely in HDFS during commit.
> Currently the files are tracked only when they are created and also getting closed during the cleanup instead of flush.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1104) HDFS rolls the first file
incorrectly
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247930#comment-13247930 ]
jiraposter@reviews.apache.org commented on FLUME-1104:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4662/
-----------------------------------------------------------
Review request for Flume.
Summary
-------
The sink process() keep tracks of the buckets opened during the transaction. At the end of transaction, we need to flush all the buckets that has pending data. This is required in order to ensure that the data removed from channel should be safely in HDFS during commit.
Currently the files are tracked only when they are created and also getting closed during the cleanup instead of flush.
The fix is to track buckets every time they are written to in the current transaction. Also buckets with pending data should be flushed instead of close.
This addresses bug FLUME-1104.
https://issues.apache.org/jira/browse/FLUME-1104
Diffs
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 114682a
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 6ff3737
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 7a94f97
Diff: https://reviews.apache.org/r/4662/diff
Testing
-------
Thanks,
Prasad
> HDFS rolls the first file incorrectly
> -------------------------------------
>
> Key: FLUME-1104
> URL: https://issues.apache.org/jira/browse/FLUME-1104
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
>
> The sink process() keep tracks of the buckets opened during the transaction. At the end of transaction, we need to flush all the buckets that has pending data. This is required in order to ensure that the data removed from channel should be safely in HDFS during commit.
> Currently the files are tracked only when they are created and also getting closed during the cleanup instead of flush.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1104) HDFS rolls the first file incorrectly
Posted by "Prasad Mujumdar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar updated FLUME-1104:
-----------------------------------
Summary: HDFS rolls the first file incorrectly (was: HDFS file roller creates causes the first file to roll too soon)
> HDFS rolls the first file incorrectly
> -------------------------------------
>
> Key: FLUME-1104
> URL: https://issues.apache.org/jira/browse/FLUME-1104
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
>
> The sink process() keep tracks of the buckets opened during the transaction. At the end of transaction, we need to flush all the buckets that has pending data. This is required in order to ensure that the data removed from channel should be safely in HDFS during commit.
> Currently the files are tracked only when they are created and also getting closed during the cleanup instead of flush.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1104) HDFS rolls the first file
incorrectly
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248007#comment-13248007 ]
jiraposter@reviews.apache.org commented on FLUME-1104:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4662/#review6731
-----------------------------------------------------------
Ship it!
+1
- Arvind
On 2012-04-06 03:06:43, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4662/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-04-06 03:06:43)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The sink process() keep tracks of the buckets opened during the transaction. At the end of transaction, we need to flush all the buckets that has pending data. This is required in order to ensure that the data removed from channel should be safely in HDFS during commit.
bq. Currently the files are tracked only when they are created and also getting closed during the cleanup instead of flush.
bq.
bq. The fix is to track buckets every time they are written to in the current transaction. Also buckets with pending data should be flushed instead of close.
bq.
bq.
bq. This addresses bug FLUME-1104.
bq. https://issues.apache.org/jira/browse/FLUME-1104
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 7a94f97
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 114682a
bq. flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 6ff3737
bq.
bq. Diff: https://reviews.apache.org/r/4662/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> HDFS rolls the first file incorrectly
> -------------------------------------
>
> Key: FLUME-1104
> URL: https://issues.apache.org/jira/browse/FLUME-1104
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Fix For: v1.2.0
>
> Attachments: FLUME-1104-2.patch
>
>
> The sink process() keep tracks of the buckets opened during the transaction. At the end of transaction, we need to flush all the buckets that has pending data. This is required in order to ensure that the data removed from channel should be safely in HDFS during commit.
> Currently the files are tracked only when they are created and also getting closed during the cleanup instead of flush.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1104) HDFS rolls the first file
incorrectly
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247951#comment-13247951 ]
jiraposter@reviews.apache.org commented on FLUME-1104:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4662/#review6726
-----------------------------------------------------------
Ship it!
+1
Please attach the patch to the jira.
- Arvind
On 2012-04-06 01:07:38, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4662/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-04-06 01:07:38)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The sink process() keep tracks of the buckets opened during the transaction. At the end of transaction, we need to flush all the buckets that has pending data. This is required in order to ensure that the data removed from channel should be safely in HDFS during commit.
bq. Currently the files are tracked only when they are created and also getting closed during the cleanup instead of flush.
bq.
bq. The fix is to track buckets every time they are written to in the current transaction. Also buckets with pending data should be flushed instead of close.
bq.
bq.
bq. This addresses bug FLUME-1104.
bq. https://issues.apache.org/jira/browse/FLUME-1104
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 114682a
bq. flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 6ff3737
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 7a94f97
bq.
bq. Diff: https://reviews.apache.org/r/4662/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> HDFS rolls the first file incorrectly
> -------------------------------------
>
> Key: FLUME-1104
> URL: https://issues.apache.org/jira/browse/FLUME-1104
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
>
> The sink process() keep tracks of the buckets opened during the transaction. At the end of transaction, we need to flush all the buckets that has pending data. This is required in order to ensure that the data removed from channel should be safely in HDFS during commit.
> Currently the files are tracked only when they are created and also getting closed during the cleanup instead of flush.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1104) HDFS rolls the first file
incorrectly
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247990#comment-13247990 ]
jiraposter@reviews.apache.org commented on FLUME-1104:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4662/
-----------------------------------------------------------
(Updated 2012-04-06 03:06:43.587885)
Review request for Flume.
Changes
-------
1) Conditionally add the bucket writer to local list.
2) The roller file extension to start with system timestamp instead of 0.
3) Updated tests that were counting number of files generated. The patch avoids the unnecessary close which reduces the file count.
Summary
-------
The sink process() keep tracks of the buckets opened during the transaction. At the end of transaction, we need to flush all the buckets that has pending data. This is required in order to ensure that the data removed from channel should be safely in HDFS during commit.
Currently the files are tracked only when they are created and also getting closed during the cleanup instead of flush.
The fix is to track buckets every time they are written to in the current transaction. Also buckets with pending data should be flushed instead of close.
This addresses bug FLUME-1104.
https://issues.apache.org/jira/browse/FLUME-1104
Diffs (updated)
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 7a94f97
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 114682a
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 6ff3737
Diff: https://reviews.apache.org/r/4662/diff
Testing
-------
Thanks,
Prasad
> HDFS rolls the first file incorrectly
> -------------------------------------
>
> Key: FLUME-1104
> URL: https://issues.apache.org/jira/browse/FLUME-1104
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: FLUME-1104-2.patch
>
>
> The sink process() keep tracks of the buckets opened during the transaction. At the end of transaction, we need to flush all the buckets that has pending data. This is required in order to ensure that the data removed from channel should be safely in HDFS during commit.
> Currently the files are tracked only when they are created and also getting closed during the cleanup instead of flush.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira