You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Brock Noland (Created) (JIRA)" <ji...@apache.org> on 2012/02/15 21:11:00 UTC
[jira] [Created] (FLUME-973) Tailing a file to HDFS does not work
as expected when datanode restarts
Tailing a file to HDFS does not work as expected when datanode restarts
-----------------------------------------------------------------------
Key: FLUME-973
URL: https://issues.apache.org/jira/browse/FLUME-973
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v1.0.0
Reporter: Brock Noland
Assignee: Brock Noland
If you have an exec source of tail -F file and an HDFSEventSink, restarting the datanode leads to flume not not writing to HDFS. However, flume continues to run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-973) Tailing a file to HDFS does not work
as expected when datanode restarts
Posted by "Brock Noland (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated FLUME-973:
-------------------------------
Attachment: FLUME-973-1.patch
Uncommented cleanup code.
> Tailing a file to HDFS does not work as expected when datanode restarts
> -----------------------------------------------------------------------
>
> Key: FLUME-973
> URL: https://issues.apache.org/jira/browse/FLUME-973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Assignee: Brock Noland
> Attachments: FLUME-973-0.patch, FLUME-973-1.patch
>
>
> If you have an exec source of tail -F file and an HDFSEventSink, restarting the datanode leads to flume not not writing to HDFS. However, flume continues to run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (FLUME-973) Tailing a file to HDFS does not work
as expected when datanode restarts
Posted by "Brock Noland (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland resolved FLUME-973.
--------------------------------
Resolution: Won't Fix
> Tailing a file to HDFS does not work as expected when datanode restarts
> -----------------------------------------------------------------------
>
> Key: FLUME-973
> URL: https://issues.apache.org/jira/browse/FLUME-973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Assignee: Brock Noland
> Attachments: FLUME-973-0.patch, FLUME-973-1.patch, FLUME-973-2.patch
>
>
> If you have an exec source of tail -F file and an HDFSEventSink, restarting the datanode leads to flume not not writing to HDFS. However, flume continues to run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-973) Tailing a file to HDFS does not work
as expected when datanode restarts
Posted by "Brock Noland (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated FLUME-973:
-------------------------------
Attachment: FLUME-973-0.patch
Attached patch adds restart option (disabled by default) to ExecSource and changes HDFSEventSink to handle the case when the datanodes are restarted.
> Tailing a file to HDFS does not work as expected when datanode restarts
> -----------------------------------------------------------------------
>
> Key: FLUME-973
> URL: https://issues.apache.org/jira/browse/FLUME-973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Assignee: Brock Noland
> Attachments: FLUME-973-0.patch
>
>
> If you have an exec source of tail -F file and an HDFSEventSink, restarting the datanode leads to flume not not writing to HDFS. However, flume continues to run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-973) Tailing a file to HDFS does not work
as expected when datanode restarts
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208989#comment-13208989 ]
jiraposter@reviews.apache.org commented on FLUME-973:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3913/
-----------------------------------------------------------
(Updated 2012-02-15 23:57:07.815781)
Review request for Flume.
Changes
-------
Added stop ExecSource$ExecRunnable when source is stopped.
Summary
-------
Attached patch adds restart option (disabled by default) to ExecSource and changes HDFSEventSink to handle the case when the datanodes are restarted.
This addresses bug FLUME-973.
https://issues.apache.org/jira/browse/FLUME-973
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
flume-ng-core/src/main/java/org/apache/flume/SinkRunner.java 074aab3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 6a17f06
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java a96016c
flume-ng-sinks/flume-hdfs-sink/pom.xml f27851e
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 45769f6
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 19b2559
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java 8a6740f
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSBadSeqWriter.java PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
Diff: https://reviews.apache.org/r/3913/diff
Testing
-------
Added tests to cover new code and tested build manually by restarting datanode.
Thanks,
Brock
> Tailing a file to HDFS does not work as expected when datanode restarts
> -----------------------------------------------------------------------
>
> Key: FLUME-973
> URL: https://issues.apache.org/jira/browse/FLUME-973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Assignee: Brock Noland
> Attachments: FLUME-973-0.patch, FLUME-973-1.patch, FLUME-973-2.patch
>
>
> If you have an exec source of tail -F file and an HDFSEventSink, restarting the datanode leads to flume not not writing to HDFS. However, flume continues to run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-973) Tailing a file to HDFS does not work
as expected when datanode restarts
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208759#comment-13208759 ]
jiraposter@reviews.apache.org commented on FLUME-973:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3913/
-----------------------------------------------------------
(Updated 2012-02-15 20:17:01.338190)
Review request for Flume.
Changes
-------
Uncommented cleanup code.
Summary
-------
Attached patch adds restart option (disabled by default) to ExecSource and changes HDFSEventSink to handle the case when the datanodes are restarted.
This addresses bug FLUME-973.
https://issues.apache.org/jira/browse/FLUME-973
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
flume-ng-core/src/main/java/org/apache/flume/SinkRunner.java 074aab3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 6a17f06
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java a96016c
flume-ng-sinks/flume-hdfs-sink/pom.xml f27851e
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 45769f6
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 19b2559
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java 8a6740f
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSBadSeqWriter.java PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
Diff: https://reviews.apache.org/r/3913/diff
Testing
-------
Added tests to cover new code and tested build manually by restarting datanode.
Thanks,
Brock
> Tailing a file to HDFS does not work as expected when datanode restarts
> -----------------------------------------------------------------------
>
> Key: FLUME-973
> URL: https://issues.apache.org/jira/browse/FLUME-973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Assignee: Brock Noland
> Attachments: FLUME-973-0.patch, FLUME-973-1.patch
>
>
> If you have an exec source of tail -F file and an HDFSEventSink, restarting the datanode leads to flume not not writing to HDFS. However, flume continues to run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-973) Tailing a file to HDFS does not work
as expected when datanode restarts
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208754#comment-13208754 ]
jiraposter@reviews.apache.org commented on FLUME-973:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3913/
-----------------------------------------------------------
Review request for Flume.
Summary
-------
Attached patch adds restart option (disabled by default) to ExecSource and changes HDFSEventSink to handle the case when the datanodes are restarted.
This addresses bug FLUME-973.
https://issues.apache.org/jira/browse/FLUME-973
Diffs
-----
flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
flume-ng-core/src/main/java/org/apache/flume/SinkRunner.java 074aab3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 6a17f06
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java a96016c
flume-ng-sinks/flume-hdfs-sink/pom.xml f27851e
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 45769f6
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 19b2559
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/HDFSBadSeqWriter.java 8a6740f
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSBadSeqWriter.java PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
Diff: https://reviews.apache.org/r/3913/diff
Testing
-------
Added tests to cover new code and tested build manually by restarting datanode.
Thanks,
Brock
> Tailing a file to HDFS does not work as expected when datanode restarts
> -----------------------------------------------------------------------
>
> Key: FLUME-973
> URL: https://issues.apache.org/jira/browse/FLUME-973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Assignee: Brock Noland
> Attachments: FLUME-973-0.patch
>
>
> If you have an exec source of tail -F file and an HDFSEventSink, restarting the datanode leads to flume not not writing to HDFS. However, flume continues to run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-973) Tailing a file to HDFS does not work
as expected when datanode restarts
Posted by "Brock Noland (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated FLUME-973:
-------------------------------
Fix Version/s: (was: v1.1.0)
Status: Open (was: Patch Available)
Patch will be split up into multiple patches/JIRAs.
> Tailing a file to HDFS does not work as expected when datanode restarts
> -----------------------------------------------------------------------
>
> Key: FLUME-973
> URL: https://issues.apache.org/jira/browse/FLUME-973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Assignee: Brock Noland
> Attachments: FLUME-973-0.patch, FLUME-973-1.patch, FLUME-973-2.patch
>
>
> If you have an exec source of tail -F file and an HDFSEventSink, restarting the datanode leads to flume not not writing to HDFS. However, flume continues to run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-973) Tailing a file to HDFS does not work
as expected when datanode restarts
Posted by "Brock Noland (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated FLUME-973:
-------------------------------
Fix Version/s: v1.1.0
Status: Patch Available (was: Open)
Marking as patch available.
> Tailing a file to HDFS does not work as expected when datanode restarts
> -----------------------------------------------------------------------
>
> Key: FLUME-973
> URL: https://issues.apache.org/jira/browse/FLUME-973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Assignee: Brock Noland
> Fix For: v1.1.0
>
> Attachments: FLUME-973-0.patch, FLUME-973-1.patch, FLUME-973-2.patch
>
>
> If you have an exec source of tail -F file and an HDFSEventSink, restarting the datanode leads to flume not not writing to HDFS. However, flume continues to run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-973) Tailing a file to HDFS does not work
as expected when datanode restarts
Posted by "Brock Noland (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13211522#comment-13211522 ]
Brock Noland commented on FLUME-973:
------------------------------------
This is an expansion of FLUME-871
> Tailing a file to HDFS does not work as expected when datanode restarts
> -----------------------------------------------------------------------
>
> Key: FLUME-973
> URL: https://issues.apache.org/jira/browse/FLUME-973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Assignee: Brock Noland
> Fix For: v1.1.0
>
> Attachments: FLUME-973-0.patch, FLUME-973-1.patch, FLUME-973-2.patch
>
>
> If you have an exec source of tail -F file and an HDFSEventSink, restarting the datanode leads to flume not not writing to HDFS. However, flume continues to run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-973) Tailing a file to HDFS does not work
as expected when datanode restarts
Posted by "Brock Noland (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated FLUME-973:
-------------------------------
Attachment: FLUME-973-2.patch
Latest patch
> Tailing a file to HDFS does not work as expected when datanode restarts
> -----------------------------------------------------------------------
>
> Key: FLUME-973
> URL: https://issues.apache.org/jira/browse/FLUME-973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Assignee: Brock Noland
> Attachments: FLUME-973-0.patch, FLUME-973-1.patch, FLUME-973-2.patch
>
>
> If you have an exec source of tail -F file and an HDFSEventSink, restarting the datanode leads to flume not not writing to HDFS. However, flume continues to run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-973) Tailing a file to HDFS does not work
as expected when datanode restarts
Posted by "Brock Noland (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212789#comment-13212789 ]
Brock Noland commented on FLUME-973:
------------------------------------
These JIRA's supersede this one:
FLUME-979 - ExecSource should optionally restart the command when it exits
FLUME-978 - Context interface is too basic requiring boilerplate user code
FLUME-984 - SinkRunner should catch unhanded exceptions and log them like PollingSourceRunner
FLUME-985 - All HDFS Operations in HDFSEventSink should have a timeout
> Tailing a file to HDFS does not work as expected when datanode restarts
> -----------------------------------------------------------------------
>
> Key: FLUME-973
> URL: https://issues.apache.org/jira/browse/FLUME-973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Assignee: Brock Noland
> Attachments: FLUME-973-0.patch, FLUME-973-1.patch, FLUME-973-2.patch
>
>
> If you have an exec source of tail -F file and an HDFSEventSink, restarting the datanode leads to flume not not writing to HDFS. However, flume continues to run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira