You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Ted Malaska <te...@cloudera.com> on 2012/09/17 17:49:44 UTC

Review Request: RollingFileSink need to be able to construct directory path based on escape sequence

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7131/
-----------------------------------------------------------

Review request for Flume.


Description
-------

OK, first note this is still a work in progress, but I'm at the point where I need feedback.

Initial goal was to allow RollingFileSink to do Escape Sequence the same way as HDFSEventSink.

In order to reuse code, my change attempt to reuse bucketWriter and BucketWriterLinkedHashMap.

BucketWriter has been broken up into three files.

> AbstractBucketWriter : Contains all the rollover and escape sequence logic.
> BucketWriter: Contains the IO logic to HDFS
> FileBucketWriter: contains the IO logic to normal files

I also added a couple unit tests to RollingFileSink for new types of roll overs and escape sequences.


This addresses bug FLUME-1295.
    https://issues.apache.org/jira/browse/FLUME-1295


Diffs
-----

  flume-ng-core/src/main/java/org/apache/flume/sink/AbstractBucketWriter.java PRE-CREATION 
  flume-ng-core/src/main/java/org/apache/flume/sink/BucketWriterLinkedHashMap.java PRE-CREATION 
  flume-ng-core/src/main/java/org/apache/flume/sink/FileBucketWriter.java PRE-CREATION 
  flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java be640bb 
  flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 07fa644 
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 6408eb9 
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9a76ecb 

Diff: https://reviews.apache.org/r/7131/diff/


Testing
-------


Thanks,

Ted Malaska


Re: Review Request: RollingFileSink need to be able to construct directory path based on escape sequence

Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7131/#review14021
-----------------------------------------------------------


Echoing my comments from the JIRA. Personally what I think we should do with this patch is not modify HDFSEventSink but get all the "infrastructure" ready so we could modify HDFSEventSink once we are sure the new code is working correctly. The reason is HDFSEventSink is used by a ton of people and it's been a source of bugs in the past.


- Brock Noland


On Sept. 17, 2012, 10:34 p.m., Ted Malaska wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7131/
> -----------------------------------------------------------
> 
> (Updated Sept. 17, 2012, 10:34 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> -------
> 
> OK, first note this is still a work in progress, but I'm at the point where I need feedback.
> 
> Initial goal was to allow RollingFileSink to do Escape Sequence the same way as HDFSEventSink.
> 
> In order to reuse code, my change attempt to reuse bucketWriter and BucketWriterLinkedHashMap.
> 
> BucketWriter has been broken up into three files.
> 
> > AbstractBucketWriter : Contains all the rollover and escape sequence logic.
> > BucketWriter: Contains the IO logic to HDFS
> > FileBucketWriter: contains the IO logic to normal files
> 
> I also added a couple unit tests to RollingFileSink for new types of roll overs and escape sequences.
> 
> 
> This addresses bug FLUME-1295.
>     https://issues.apache.org/jira/browse/FLUME-1295
> 
> 
> Diffs
> -----
> 
>   flume-ng-core/src/main/java/org/apache/flume/sink/AbstractBucketWriter.java PRE-CREATION 
>   flume-ng-core/src/main/java/org/apache/flume/sink/BucketWriterLinkedHashMap.java PRE-CREATION 
>   flume-ng-core/src/main/java/org/apache/flume/sink/FileBucketWriter.java PRE-CREATION 
>   flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java be640bb 
>   flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 07fa644 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 6408eb9 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9a76ecb 
> 
> Diff: https://reviews.apache.org/r/7131/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Ted Malaska
> 
>


Re: Review Request: RollingFileSink need to be able to construct directory path based on escape sequence

Posted by Ted Malaska <te...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7131/
-----------------------------------------------------------

(Updated Sept. 17, 2012, 10:34 p.m.)


Review request for Flume.


Changes
-------

Applied all of Brock's findings.


Description
-------

OK, first note this is still a work in progress, but I'm at the point where I need feedback.

Initial goal was to allow RollingFileSink to do Escape Sequence the same way as HDFSEventSink.

In order to reuse code, my change attempt to reuse bucketWriter and BucketWriterLinkedHashMap.

BucketWriter has been broken up into three files.

> AbstractBucketWriter : Contains all the rollover and escape sequence logic.
> BucketWriter: Contains the IO logic to HDFS
> FileBucketWriter: contains the IO logic to normal files

I also added a couple unit tests to RollingFileSink for new types of roll overs and escape sequences.


This addresses bug FLUME-1295.
    https://issues.apache.org/jira/browse/FLUME-1295


Diffs (updated)
-----

  flume-ng-core/src/main/java/org/apache/flume/sink/AbstractBucketWriter.java PRE-CREATION 
  flume-ng-core/src/main/java/org/apache/flume/sink/BucketWriterLinkedHashMap.java PRE-CREATION 
  flume-ng-core/src/main/java/org/apache/flume/sink/FileBucketWriter.java PRE-CREATION 
  flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java be640bb 
  flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 07fa644 
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 6408eb9 
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9a76ecb 

Diff: https://reviews.apache.org/r/7131/diff/


Testing
-------


Thanks,

Ted Malaska


Re: Review Request: RollingFileSink need to be able to construct directory path based on escape sequence

Posted by Ted Malaska <te...@cloudera.com>.

> On Sept. 17, 2012, 5:58 p.m., Brock Noland wrote:
> > flume-ng-core/src/main/java/org/apache/flume/sink/AbstractBucketWriter.java, line 84
> > <https://reviews.apache.org/r/7131/diff/1/?file=155840#file155840line84>
> >
> >     This is a hadoop issue and file systems which other sinks such as Rolling File sink won't have such an issue. Should this code in a class related to hadoop?

agreed


> On Sept. 17, 2012, 5:58 p.m., Brock Noland wrote:
> > flume-ng-core/src/main/java/org/apache/flume/sink/FileBucketWriter.java, line 41
> > <https://reviews.apache.org/r/7131/diff/1/?file=155842#file155842line41>
> >
> >     I think we should be using a File object to get the directory here.

agreed


> On Sept. 17, 2012, 5:58 p.m., Brock Noland wrote:
> > flume-ng-core/src/main/java/org/apache/flume/sink/FileBucketWriter.java, line 42
> > <https://reviews.apache.org/r/7131/diff/1/?file=155842#file155842line42>
> >
> >     We should do an assertion or log something if mkdirs fails.

agreed


- Ted


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7131/#review11619
-----------------------------------------------------------


On Sept. 17, 2012, 3:49 p.m., Ted Malaska wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7131/
> -----------------------------------------------------------
> 
> (Updated Sept. 17, 2012, 3:49 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> -------
> 
> OK, first note this is still a work in progress, but I'm at the point where I need feedback.
> 
> Initial goal was to allow RollingFileSink to do Escape Sequence the same way as HDFSEventSink.
> 
> In order to reuse code, my change attempt to reuse bucketWriter and BucketWriterLinkedHashMap.
> 
> BucketWriter has been broken up into three files.
> 
> > AbstractBucketWriter : Contains all the rollover and escape sequence logic.
> > BucketWriter: Contains the IO logic to HDFS
> > FileBucketWriter: contains the IO logic to normal files
> 
> I also added a couple unit tests to RollingFileSink for new types of roll overs and escape sequences.
> 
> 
> This addresses bug FLUME-1295.
>     https://issues.apache.org/jira/browse/FLUME-1295
> 
> 
> Diffs
> -----
> 
>   flume-ng-core/src/main/java/org/apache/flume/sink/AbstractBucketWriter.java PRE-CREATION 
>   flume-ng-core/src/main/java/org/apache/flume/sink/BucketWriterLinkedHashMap.java PRE-CREATION 
>   flume-ng-core/src/main/java/org/apache/flume/sink/FileBucketWriter.java PRE-CREATION 
>   flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java be640bb 
>   flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 07fa644 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 6408eb9 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9a76ecb 
> 
> Diff: https://reviews.apache.org/r/7131/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Ted Malaska
> 
>


Re: Review Request: RollingFileSink need to be able to construct directory path based on escape sequence

Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7131/#review11619
-----------------------------------------------------------


Generally I think this is a good plan. This is a core part of the Flume functionality and it has taken a fair amount of work to work around all the various hadoop kerberos items so I am hopeful that someone who has spent more time in this code can review the change.

I have provided my initial feedback below.


flume-ng-core/src/main/java/org/apache/flume/sink/AbstractBucketWriter.java
<https://reviews.apache.org/r/7131/#comment25042>

    This is a hadoop issue and file systems which other sinks such as Rolling File sink won't have such an issue. Should this code in a class related to hadoop?



flume-ng-core/src/main/java/org/apache/flume/sink/FileBucketWriter.java
<https://reviews.apache.org/r/7131/#comment25040>

    I think we should be using a File object to get the directory here.



flume-ng-core/src/main/java/org/apache/flume/sink/FileBucketWriter.java
<https://reviews.apache.org/r/7131/#comment25041>

    We should do an assertion or log something if mkdirs fails.


- Brock Noland


On Sept. 17, 2012, 3:49 p.m., Ted Malaska wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7131/
> -----------------------------------------------------------
> 
> (Updated Sept. 17, 2012, 3:49 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> -------
> 
> OK, first note this is still a work in progress, but I'm at the point where I need feedback.
> 
> Initial goal was to allow RollingFileSink to do Escape Sequence the same way as HDFSEventSink.
> 
> In order to reuse code, my change attempt to reuse bucketWriter and BucketWriterLinkedHashMap.
> 
> BucketWriter has been broken up into three files.
> 
> > AbstractBucketWriter : Contains all the rollover and escape sequence logic.
> > BucketWriter: Contains the IO logic to HDFS
> > FileBucketWriter: contains the IO logic to normal files
> 
> I also added a couple unit tests to RollingFileSink for new types of roll overs and escape sequences.
> 
> 
> This addresses bug FLUME-1295.
>     https://issues.apache.org/jira/browse/FLUME-1295
> 
> 
> Diffs
> -----
> 
>   flume-ng-core/src/main/java/org/apache/flume/sink/AbstractBucketWriter.java PRE-CREATION 
>   flume-ng-core/src/main/java/org/apache/flume/sink/BucketWriterLinkedHashMap.java PRE-CREATION 
>   flume-ng-core/src/main/java/org/apache/flume/sink/FileBucketWriter.java PRE-CREATION 
>   flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java be640bb 
>   flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 07fa644 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 6408eb9 
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9a76ecb 
> 
> Diff: https://reviews.apache.org/r/7131/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Ted Malaska
> 
>