You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Brock Noland <br...@cloudera.com> on 2013/01/15 23:27:18 UTC

Review Request: FLUME-1844: HDFSEventSink should have option to use RawLocalFileSystem

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8964/
-----------------------------------------------------------

Review request for Flume.


Description
-------

Add's flag to HDFSEventSink Serializer's to use raw local file system.


This addresses bug FLUME-1844.
    https://issues.apache.org/jira/browse/FLUME-1844


Diffs
-----

  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java f14f7cb 
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 18fe6d4 
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java bd40a88 
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1e6d68f 
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 1035ac3 
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java PRE-CREATION 

Diff: https://reviews.apache.org/r/8964/diff/


Testing
-------

Unit tests pass.


Thanks,

Brock Noland


Re: Review Request: FLUME-1844: HDFSEventSink should have option to use RawLocalFileSystem

Posted by Hari Shreedharan <hs...@cloudera.com>.

> On Jan. 16, 2013, 12:21 a.m., Hari Shreedharan wrote:
> > Brock,
> > 
> > Looks good in general. One question I have is if we could just do without a flume flag. Rather check the Configuration object to see if it is set to write to local file system. Does that make sense?
> 
> Brock Noland wrote:
>     I thought about that as well. The reason I did not do that is because file:/// implements ChecksumFileSystem so data you read/write is checksummed. We could probably change it and no one would notice but it's possible someone is using this feature.... Thoughts?

Makes sense. +1


- Hari


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8964/#review15361
-----------------------------------------------------------


On Jan. 15, 2013, 10:33 p.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8964/
> -----------------------------------------------------------
> 
> (Updated Jan. 15, 2013, 10:33 p.m.)
> 
> 
> Review request for Flume, Juhani Connolly and Hari Shreedharan.
> 
> 
> Description
> -------
> 
> Add's flag to HDFSEventSink Serializer's to use raw local file system.
> 
> 
> This addresses bug FLUME-1844.
>     https://issues.apache.org/jira/browse/FLUME-1844
> 
> 
> Diffs
> -----
> 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java f14f7cb 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 18fe6d4 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java bd40a88 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1e6d68f 
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 1035ac3 
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/8964/diff/
> 
> 
> Testing
> -------
> 
> Unit tests pass.
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Re: Review Request: FLUME-1844: HDFSEventSink should have option to use RawLocalFileSystem

Posted by Brock Noland <br...@cloudera.com>.

> On Jan. 16, 2013, 12:21 a.m., Hari Shreedharan wrote:
> > Brock,
> > 
> > Looks good in general. One question I have is if we could just do without a flume flag. Rather check the Configuration object to see if it is set to write to local file system. Does that make sense?

I thought about that as well. The reason I did not do that is because file:/// implements ChecksumFileSystem so data you read/write is checksummed. We could probably change it and no one would notice but it's possible someone is using this feature.... Thoughts?


- Brock


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8964/#review15361
-----------------------------------------------------------


On Jan. 15, 2013, 10:33 p.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8964/
> -----------------------------------------------------------
> 
> (Updated Jan. 15, 2013, 10:33 p.m.)
> 
> 
> Review request for Flume, Juhani Connolly and Hari Shreedharan.
> 
> 
> Description
> -------
> 
> Add's flag to HDFSEventSink Serializer's to use raw local file system.
> 
> 
> This addresses bug FLUME-1844.
>     https://issues.apache.org/jira/browse/FLUME-1844
> 
> 
> Diffs
> -----
> 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java f14f7cb 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 18fe6d4 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java bd40a88 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1e6d68f 
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 1035ac3 
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/8964/diff/
> 
> 
> Testing
> -------
> 
> Unit tests pass.
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Re: Review Request: FLUME-1844: HDFSEventSink should have option to use RawLocalFileSystem

Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8964/#review15361
-----------------------------------------------------------


Brock,

Looks good in general. One question I have is if we could just do without a flume flag. Rather check the Configuration object to see if it is set to write to local file system. Does that make sense?

- Hari Shreedharan


On Jan. 15, 2013, 10:33 p.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8964/
> -----------------------------------------------------------
> 
> (Updated Jan. 15, 2013, 10:33 p.m.)
> 
> 
> Review request for Flume, Juhani Connolly and Hari Shreedharan.
> 
> 
> Description
> -------
> 
> Add's flag to HDFSEventSink Serializer's to use raw local file system.
> 
> 
> This addresses bug FLUME-1844.
>     https://issues.apache.org/jira/browse/FLUME-1844
> 
> 
> Diffs
> -----
> 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java f14f7cb 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 18fe6d4 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java bd40a88 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1e6d68f 
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 1035ac3 
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/8964/diff/
> 
> 
> Testing
> -------
> 
> Unit tests pass.
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Re: Review Request: FLUME-1844: HDFSEventSink should have option to use RawLocalFileSystem

Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8964/
-----------------------------------------------------------

(Updated Jan. 15, 2013, 10:33 p.m.)


Review request for Flume, Juhani Connolly and Hari Shreedharan.


Changes
-------

Added specific reviewers.


Description
-------

Add's flag to HDFSEventSink Serializer's to use raw local file system.


This addresses bug FLUME-1844.
    https://issues.apache.org/jira/browse/FLUME-1844


Diffs
-----

  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java f14f7cb 
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 18fe6d4 
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java bd40a88 
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1e6d68f 
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 1035ac3 
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java PRE-CREATION 

Diff: https://reviews.apache.org/r/8964/diff/


Testing
-------

Unit tests pass.


Thanks,

Brock Noland