You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Inder Singh <in...@gmail.com> on 2012/04/03 15:05:08 UTC

Review Request: Patch addressing Issues in using AvroClient with HDFSEventSink for regex based destination paths

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/
-----------------------------------------------------------

Review request for Flume, Arvind Prabhakar and Mike Percy.


Summary
-------

Patch to address FLUME 1096. Please review and share your thoughts.


This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
    https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096


Diffs
-----

  trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1308726 

Diff: https://reviews.apache.org/r/4622/diff


Testing
-------

1. Configured flume agent to publish data in /data/flume/%D
2. ran the avroClient to stream some data.
3. Things looked ok. Data was available in HDFS.


Thanks,

Inder

Re: Review Request: Patch addressing Issues in using AvroClient with HDFSEventSink for regex based destination paths

Posted by Inder Singh <in...@gmail.com>.

> On 2012-04-20 00:42:42, Hari Shreedharan wrote:
> > trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java, line 85
> > <https://reviews.apache.org/r/4622/diff/3/?file=101043#file101043line85>
> >
> >     This a minor comment. Doing this would make "," a delimiter in the headers and would not allow a user to use a "," in the event header text. Maybe allow an alternate way to add headers, from a file, maybe a properties file or something,when the user cannot use "," as a delim. 
> >     
> >     Also maybe convert the command line input into a properties by replacing the ","  with "\n" and wrapping it in a input stream to handle the case of rogue spaces? 
> >     
> >     You could do something like:
> >     
> >     private void parseHeaders(Properties p) //takes the properties and puts them into the headers
> >     
> >     and
> >     
> >     private void parseCommandLineHeaders(String hdrs){
> >       hdrs.replaceAll(",", "\n");
> >     //Wrap this hdrs in an InputStream,and load a properties object using that
> >       parseHeaders(Properties p);
> >     }
> >     
> >
> 
> Hari Shreedharan wrote:
>     Just to clarify: this was a nit,a good-to-have, not exactly necessary to have. Also any alternate way of allowing "," etc would be ok, this was just a suggestion. The headers I am talking about are something like flume.root.logger=DEBUG,console(just an example)

Sure, headers coming through a property seems better. I'll re-work and re-post for review.


- Inder


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/#review7054
-----------------------------------------------------------


On 2012-04-09 09:52:21, Inder Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4622/
> -----------------------------------------------------------
> 
> (Updated 2012-04-09 09:52:21)
> 
> 
> Review request for Flume, Arvind Prabhakar and Mike Percy.
> 
> 
> Summary
> -------
> 
> Patch to address FLUME 1096. Please review and share your thoughts.
> 
> 
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096
> 
> 
> Diffs
> -----
> 
>   trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1311132 
>   trunk/bin/flume-ng 1311132 
> 
> Diff: https://reviews.apache.org/r/4622/diff
> 
> 
> Testing
> -------
> 
> 1. Configured flume agent to publish data in /data/flume/%D
> 2. ran the avroClient to stream some data.
> 3. Things looked ok. Data was available in HDFS.
> 
> 
> Thanks,
> 
> Inder
> 
>

Re: Review Request: Patch addressing Issues in using AvroClient with HDFSEventSink for regex based destination paths

Posted by Hari Shreedharan <hs...@cloudera.com>.

> On 2012-04-20 00:42:42, Hari Shreedharan wrote:
> > trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java, line 85
> > <https://reviews.apache.org/r/4622/diff/3/?file=101043#file101043line85>
> >
> >     This a minor comment. Doing this would make "," a delimiter in the headers and would not allow a user to use a "," in the event header text. Maybe allow an alternate way to add headers, from a file, maybe a properties file or something,when the user cannot use "," as a delim. 
> >     
> >     Also maybe convert the command line input into a properties by replacing the ","  with "\n" and wrapping it in a input stream to handle the case of rogue spaces? 
> >     
> >     You could do something like:
> >     
> >     private void parseHeaders(Properties p) //takes the properties and puts them into the headers
> >     
> >     and
> >     
> >     private void parseCommandLineHeaders(String hdrs){
> >       hdrs.replaceAll(",", "\n");
> >     //Wrap this hdrs in an InputStream,and load a properties object using that
> >       parseHeaders(Properties p);
> >     }
> >     
> >

Just to clarify: this was a nit,a good-to-have, not exactly necessary to have. Also any alternate way of allowing "," etc would be ok, this was just a suggestion. The headers I am talking about are something like flume.root.logger=DEBUG,console(just an example) 


- Hari


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/#review7054
-----------------------------------------------------------


On 2012-04-09 09:52:21, Inder Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4622/
> -----------------------------------------------------------
> 
> (Updated 2012-04-09 09:52:21)
> 
> 
> Review request for Flume, Arvind Prabhakar and Mike Percy.
> 
> 
> Summary
> -------
> 
> Patch to address FLUME 1096. Please review and share your thoughts.
> 
> 
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096
> 
> 
> Diffs
> -----
> 
>   trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1311132 
>   trunk/bin/flume-ng 1311132 
> 
> Diff: https://reviews.apache.org/r/4622/diff
> 
> 
> Testing
> -------
> 
> 1. Configured flume agent to publish data in /data/flume/%D
> 2. ran the avroClient to stream some data.
> 3. Things looked ok. Data was available in HDFS.
> 
> 
> Thanks,
> 
> Inder
> 
>

Re: Review Request: Patch addressing Issues in using AvroClient with HDFSEventSink for regex based destination paths

Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/#review7054
-----------------------------------------------------------


Looks good! I have one minor suggestion, posted it inline.


trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
<https://reviews.apache.org/r/4622/#comment15648>

    This a minor comment. Doing this would make "," a delimiter in the headers and would not allow a user to use a "," in the event header text. Maybe allow an alternate way to add headers, from a file, maybe a properties file or something,when the user cannot use "," as a delim. 
    
    Also maybe convert the command line input into a properties by replacing the ","  with "\n" and wrapping it in a input stream to handle the case of rogue spaces? 
    
    You could do something like:
    
    private void parseHeaders(Properties p) //takes the properties and puts them into the headers
    
    and
    
    private void parseCommandLineHeaders(String hdrs){
      hdrs.replaceAll(",", "\n");
    //Wrap this hdrs in an InputStream,and load a properties object using that
      parseHeaders(Properties p);
    }
    
    


- Hari


On 2012-04-09 09:52:21, Inder Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4622/
> -----------------------------------------------------------
> 
> (Updated 2012-04-09 09:52:21)
> 
> 
> Review request for Flume, Arvind Prabhakar and Mike Percy.
> 
> 
> Summary
> -------
> 
> Patch to address FLUME 1096. Please review and share your thoughts.
> 
> 
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096
> 
> 
> Diffs
> -----
> 
>   trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1311132 
>   trunk/bin/flume-ng 1311132 
> 
> Diff: https://reviews.apache.org/r/4622/diff
> 
> 
> Testing
> -------
> 
> 1. Configured flume agent to publish data in /data/flume/%D
> 2. ran the avroClient to stream some data.
> 3. Things looked ok. Data was available in HDFS.
> 
> 
> Thanks,
> 
> Inder
> 
>

Re: Review Request: Patch addressing Issues in using AvroClient with HDFSEventSink for regex based destination paths

Posted by Mike Percy <mp...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/#review7052
-----------------------------------------------------------


Looks good. Thanks for the patch!

Nit: please change the name of the JIRA to something like: "support passing headers in AvroCLIClient". Right now the title doesn't match up very well with the content of the patch.


trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
<https://reviews.apache.org/r/4622/#comment15647>

    Nit: is this valid syntax in the --help output? Shouldn't the space after the "," be removed?


- Mike


On 2012-04-09 09:52:21, Inder Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4622/
> -----------------------------------------------------------
> 
> (Updated 2012-04-09 09:52:21)
> 
> 
> Review request for Flume, Arvind Prabhakar and Mike Percy.
> 
> 
> Summary
> -------
> 
> Patch to address FLUME 1096. Please review and share your thoughts.
> 
> 
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096
> 
> 
> Diffs
> -----
> 
>   trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1311132 
>   trunk/bin/flume-ng 1311132 
> 
> Diff: https://reviews.apache.org/r/4622/diff
> 
> 
> Testing
> -------
> 
> 1. Configured flume agent to publish data in /data/flume/%D
> 2. ran the avroClient to stream some data.
> 3. Things looked ok. Data was available in HDFS.
> 
> 
> Thanks,
> 
> Inder
> 
>

Re: Review Request: Patch adding support to pass headers through AvroCLIClient

Posted by Arvind Prabhakar <ar...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/#review7122
-----------------------------------------------------------


Thanks for the patch Inder, changes look good Inder. Some minor feedback below. 


trunk/bin/flume-ng
<https://reviews.apache.org/r/4622/#comment15739>

    Please make sure you are space/indent aligned. Also suggest using <file> instead of <headerFile>  for brevity:
    
    --headerFile,-R <file>  text file containing headers
    
    



trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
<https://reviews.apache.org/r/4622/#comment15737>

    convention: catch on the same line as closing brace:
     ...
    } catch (Exception e) {
     ...



trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
<https://reviews.apache.org/r/4622/#comment15738>

    convention: terminating paranthesis should be on the same line. If line length is more than 80 char, please use a shorter name for the entry or consider wrapping the line into two lines.


- Arvind


On 2012-04-20 08:50:23, Inder Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4622/
> -----------------------------------------------------------
> 
> (Updated 2012-04-20 08:50:23)
> 
> 
> Review request for Flume, Arvind Prabhakar and Mike Percy.
> 
> 
> Summary
> -------
> 
> Patch to address FLUME 1096. Please review and share your thoughts.
> 
> 
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096
> 
> 
> Diffs
> -----
> 
>   trunk/bin/flume-ng 1328221 
>   trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1328221 
> 
> Diff: https://reviews.apache.org/r/4622/diff
> 
> 
> Testing
> -------
> 
> 1. Configured flume agent to publish data in /data/flume/%D
> 2. ran the avroClient to stream some data.
> 3. Things looked ok. Data was available in HDFS.
> 
> 
> Thanks,
> 
> Inder
> 
>

Re: Review Request: Patch adding support to pass headers through AvroCLIClient

Posted by Inder Singh <in...@gmail.com>.

> On 2012-04-20 17:05:03, Hari Shreedharan wrote:
> > Looks good!
> 
> Inder Singh wrote:
>     Hari,
>     
>     The patch is also available at - https://issues.apache.org/jira/browse/FLUME-1096 ->https://issues.apache.org/jira/secure/attachment/12523468/FLUME-1096-4.patch for you download and commit.
> 
> Hari Shreedharan wrote:
>     I am not a committer. You will need to wait for one of the committers to review.

Sure :)


- Inder


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/#review7082
-----------------------------------------------------------


On 2012-04-20 08:50:23, Inder Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4622/
> -----------------------------------------------------------
> 
> (Updated 2012-04-20 08:50:23)
> 
> 
> Review request for Flume, Arvind Prabhakar and Mike Percy.
> 
> 
> Summary
> -------
> 
> Patch to address FLUME 1096. Please review and share your thoughts.
> 
> 
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096
> 
> 
> Diffs
> -----
> 
>   trunk/bin/flume-ng 1328221 
>   trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1328221 
> 
> Diff: https://reviews.apache.org/r/4622/diff
> 
> 
> Testing
> -------
> 
> 1. Configured flume agent to publish data in /data/flume/%D
> 2. ran the avroClient to stream some data.
> 3. Things looked ok. Data was available in HDFS.
> 
> 
> Thanks,
> 
> Inder
> 
>

Re: Review Request: Patch adding support to pass headers through AvroCLIClient

Posted by Hari Shreedharan <hs...@cloudera.com>.

> On 2012-04-20 17:05:03, Hari Shreedharan wrote:
> > Looks good!
> 
> Inder Singh wrote:
>     Hari,
>     
>     The patch is also available at - https://issues.apache.org/jira/browse/FLUME-1096 ->https://issues.apache.org/jira/secure/attachment/12523468/FLUME-1096-4.patch for you download and commit.

I am not a committer. You will need to wait for one of the committers to review.


- Hari


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/#review7082
-----------------------------------------------------------


On 2012-04-20 08:50:23, Inder Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4622/
> -----------------------------------------------------------
> 
> (Updated 2012-04-20 08:50:23)
> 
> 
> Review request for Flume, Arvind Prabhakar and Mike Percy.
> 
> 
> Summary
> -------
> 
> Patch to address FLUME 1096. Please review and share your thoughts.
> 
> 
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096
> 
> 
> Diffs
> -----
> 
>   trunk/bin/flume-ng 1328221 
>   trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1328221 
> 
> Diff: https://reviews.apache.org/r/4622/diff
> 
> 
> Testing
> -------
> 
> 1. Configured flume agent to publish data in /data/flume/%D
> 2. ran the avroClient to stream some data.
> 3. Things looked ok. Data was available in HDFS.
> 
> 
> Thanks,
> 
> Inder
> 
>

Re: Review Request: Patch adding support to pass headers through AvroCLIClient

Posted by Inder Singh <in...@gmail.com>.

> On 2012-04-20 17:05:03, Hari Shreedharan wrote:
> > Looks good!

Hari,

The patch is also available at - https://issues.apache.org/jira/browse/FLUME-1096 ->https://issues.apache.org/jira/secure/attachment/12523468/FLUME-1096-4.patch for you download and commit.


- Inder


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/#review7082
-----------------------------------------------------------


On 2012-04-20 08:50:23, Inder Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4622/
> -----------------------------------------------------------
> 
> (Updated 2012-04-20 08:50:23)
> 
> 
> Review request for Flume, Arvind Prabhakar and Mike Percy.
> 
> 
> Summary
> -------
> 
> Patch to address FLUME 1096. Please review and share your thoughts.
> 
> 
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096
> 
> 
> Diffs
> -----
> 
>   trunk/bin/flume-ng 1328221 
>   trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1328221 
> 
> Diff: https://reviews.apache.org/r/4622/diff
> 
> 
> Testing
> -------
> 
> 1. Configured flume agent to publish data in /data/flume/%D
> 2. ran the avroClient to stream some data.
> 3. Things looked ok. Data was available in HDFS.
> 
> 
> Thanks,
> 
> Inder
> 
>

Re: Review Request: Patch adding support to pass headers through AvroCLIClient

Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/#review7082
-----------------------------------------------------------

Ship it!


Looks good!

- Hari


On 2012-04-20 08:50:23, Inder Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4622/
> -----------------------------------------------------------
> 
> (Updated 2012-04-20 08:50:23)
> 
> 
> Review request for Flume, Arvind Prabhakar and Mike Percy.
> 
> 
> Summary
> -------
> 
> Patch to address FLUME 1096. Please review and share your thoughts.
> 
> 
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096
> 
> 
> Diffs
> -----
> 
>   trunk/bin/flume-ng 1328221 
>   trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1328221 
> 
> Diff: https://reviews.apache.org/r/4622/diff
> 
> 
> Testing
> -------
> 
> 1. Configured flume agent to publish data in /data/flume/%D
> 2. ran the avroClient to stream some data.
> 3. Things looked ok. Data was available in HDFS.
> 
> 
> Thanks,
> 
> Inder
> 
>

Re: Review Request: Patch adding support to pass headers through AvroCLIClient

Posted by Arvind Prabhakar <ar...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/#review7134
-----------------------------------------------------------



trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
<https://reviews.apache.org/r/4622/#comment15765>

    My bad Inder - just noticed this issue that I should have caught in the first round of review. The FileInputStream is not getting closed correctly/timely. Please refactor this so that the file input stream is correctly getting closed in the finally clause associated with the try/catch.


- Arvind


On 2012-04-23 05:48:30, Inder Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4622/
> -----------------------------------------------------------
> 
> (Updated 2012-04-23 05:48:30)
> 
> 
> Review request for Flume, Arvind Prabhakar and Mike Percy.
> 
> 
> Summary
> -------
> 
> Patch to address FLUME 1096. Please review and share your thoughts.
> 
> 
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096
> 
> 
> Diffs
> -----
> 
>   trunk/bin/flume-ng 1328221 
>   trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1328221 
> 
> Diff: https://reviews.apache.org/r/4622/diff
> 
> 
> Testing
> -------
> 
> 1. Configured flume agent to publish data in /data/flume/%D
> 2. ran the avroClient to stream some data.
> 3. Things looked ok. Data was available in HDFS.
> 
> 
> Thanks,
> 
> Inder
> 
>

Re: Review Request: Patch adding support to pass headers through AvroCLIClient

Posted by Arvind Prabhakar <ar...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/#review7157
-----------------------------------------------------------

Ship it!


Looks good Inder. Some minor nits noted below. Once you address them, please attach the updated patch to the Jira.


trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
<https://reviews.apache.org/r/4622/#comment15799>

    better log the exception as well:
    
    logger.debug("Unable to load file: " + headerFile, e);



trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
<https://reviews.apache.org/r/4622/#comment15798>

    convention nit: finally on the same line as the preceding curly brace:
    
    ...
    } finally {
    ...
    }



trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
<https://reviews.apache.org/r/4622/#comment15800>

    better to log the exception here stating unable to close file.


- Arvind


On 2012-04-23 18:09:31, Inder Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4622/
> -----------------------------------------------------------
> 
> (Updated 2012-04-23 18:09:31)
> 
> 
> Review request for Flume, Arvind Prabhakar and Mike Percy.
> 
> 
> Summary
> -------
> 
> Patch to address FLUME 1096. Please review and share your thoughts.
> 
> 
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096
> 
> 
> Diffs
> -----
> 
>   trunk/bin/flume-ng 1329352 
>   trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1329352 
> 
> Diff: https://reviews.apache.org/r/4622/diff
> 
> 
> Testing
> -------
> 
> 1. Configured flume agent to publish data in /data/flume/%D
> 2. ran the avroClient to stream some data.
> 3. Things looked ok. Data was available in HDFS.
> 
> 
> Thanks,
> 
> Inder
> 
>

Re: Review Request: Patch adding support to pass headers through AvroCLIClient

Posted by Arvind Prabhakar <ar...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/#review7171
-----------------------------------------------------------

Ship it!


+1

- Arvind


On 2012-04-24 05:21:24, Inder Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4622/
> -----------------------------------------------------------
> 
> (Updated 2012-04-24 05:21:24)
> 
> 
> Review request for Flume, Arvind Prabhakar and Mike Percy.
> 
> 
> Summary
> -------
> 
> Patch to address FLUME 1096. Please review and share your thoughts.
> 
> 
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096
> 
> 
> Diffs
> -----
> 
>   trunk/bin/flume-ng 1329554 
>   trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1329554 
> 
> Diff: https://reviews.apache.org/r/4622/diff
> 
> 
> Testing
> -------
> 
> 1. Configured flume agent to publish data in /data/flume/%D
> 2. ran the avroClient to stream some data.
> 3. Things looked ok. Data was available in HDFS.
> 
> 
> Thanks,
> 
> Inder
> 
>

Re: Review Request: Patch adding support to pass headers through AvroCLIClient

Posted by Inder Pall <in...@gmail.com>.
Arvind,

https://issues.apache.org/jira/secure/attachment/12523923/FLUME-1096-7.patchis
the final version.

Thanks,
- Inder

On Tue, Apr 24, 2012 at 10:51 AM, Inder Singh <in...@gmail.com> wrote:

>    This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4622/
>   Review request for Flume, Arvind Prabhakar and Mike Percy.
> By Inder Singh.
>
> *Updated 2012-04-24 05:21:24.677346*
> Changes
>
> 1. Logging exceptions
> 2. fixing indentation of finally clause
>
>   Description
>
> Patch to address FLUME 1096. Please review and share your thoughts.
>
>   Testing
>
> 1. Configured flume agent to publish data in /data/flume/%D
> 2. ran the avroClient to stream some data.
> 3. Things looked ok. Data was available in HDFS.
>
>
>   *Bugs: * https://issues.apache.org/jira/browse/FLUME-1096<https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096>
> Diffs (updated)
>
>    - trunk/bin/flume-ng (1329554)
>    - trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
>    (1329554)
>
> View Diff <https://reviews.apache.org/r/4622/diff/>
>



-- 
Thanks,
- Inder
  Tech Platforms @Inmobi
  Linkedin - http://goo.gl/eR4Ub

Re: Review Request: Patch adding support to pass headers through AvroCLIClient

Posted by Inder Singh <in...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/
-----------------------------------------------------------

(Updated 2012-04-24 05:21:24.677346)


Review request for Flume, Arvind Prabhakar and Mike Percy.


Changes
-------

1. Logging exceptions
2. fixing indentation of finally clause


Summary
-------

Patch to address FLUME 1096. Please review and share your thoughts.


This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
    https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096


Diffs (updated)
-----

  trunk/bin/flume-ng 1329554 
  trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1329554 

Diff: https://reviews.apache.org/r/4622/diff


Testing
-------

1. Configured flume agent to publish data in /data/flume/%D
2. ran the avroClient to stream some data.
3. Things looked ok. Data was available in HDFS.


Thanks,

Inder

Re: Review Request: Patch adding support to pass headers through AvroCLIClient

Posted by Inder Singh <in...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/
-----------------------------------------------------------

(Updated 2012-04-23 18:09:31.048216)


Review request for Flume, Arvind Prabhakar and Mike Percy.


Changes
-------

Nice catch Arvind and silly miss on my part.

Refactored and retested.


Summary
-------

Patch to address FLUME 1096. Please review and share your thoughts.


This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
    https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096


Diffs (updated)
-----

  trunk/bin/flume-ng 1329352 
  trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1329352 

Diff: https://reviews.apache.org/r/4622/diff


Testing
-------

1. Configured flume agent to publish data in /data/flume/%D
2. ran the avroClient to stream some data.
3. Things looked ok. Data was available in HDFS.


Thanks,

Inder

Re: Review Request: Patch adding support to pass headers through AvroCLIClient

Posted by Inder Singh <in...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/
-----------------------------------------------------------

(Updated 2012-04-23 05:48:30.767512)


Review request for Flume, Arvind Prabhakar and Mike Percy.


Changes
-------

Incorporated feedback from Arvind

1. indentation in flume-ng
2. catch Exception to begin on the same line
3. renamed objectObjectEntry to shorter relevant name "propertiesEntry"


Summary
-------

Patch to address FLUME 1096. Please review and share your thoughts.


This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
    https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096


Diffs (updated)
-----

  trunk/bin/flume-ng 1328221 
  trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1328221 

Diff: https://reviews.apache.org/r/4622/diff


Testing
-------

1. Configured flume agent to publish data in /data/flume/%D
2. ran the avroClient to stream some data.
3. Things looked ok. Data was available in HDFS.


Thanks,

Inder

Re: Review Request: Patch adding support to pass headers through AvroCLIClient

Posted by Inder Singh <in...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/
-----------------------------------------------------------

(Updated 2012-04-20 08:50:23.047155)


Review request for Flume, Arvind Prabhakar and Mike Percy.


Summary (updated)
-------

Patch to address FLUME 1096. Please review and share your thoughts.


This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
    https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096


Diffs
-----

  trunk/bin/flume-ng 1328221 
  trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1328221 

Diff: https://reviews.apache.org/r/4622/diff


Testing
-------

1. Configured flume agent to publish data in /data/flume/%D
2. ran the avroClient to stream some data.
3. Things looked ok. Data was available in HDFS.


Thanks,

Inder

Re: Review Request: Patch addressing Issues in using AvroClient with HDFSEventSink for regex based destination paths

Posted by Inder Pall <in...@gmail.com>.
Folks,

Updated the client to read headers from a property file.
Sample header.properties
------------------------------------
streamName = sample-stream
timestamp=2189128912

Client command line looks like ->
 ./flume-ng avro-client --conf ../conf/ -H localhost -p 41414 -F
~/.bash_profile -R headers.properties

Thanks,
- Inder

On Fri, Apr 20, 2012 at 2:11 PM, Inder Singh <in...@gmail.com> wrote:

>    This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4622/
>   Review request for Flume, Arvind Prabhakar and Mike Percy.
> By Inder Singh.
>
> *Updated 2012-04-20 08:41:44.370859*
> Changes
>
> Incorporated the following review comments
>
> 1. header's come through a config file.
> Sample header.properties
> streamName=sample-test-stream
> timestamp=2329329399
>
> Agent Command Line looks like ->
> ./flume-ng avro-client --conf ../conf/ -H localhost -p 41414 -F ~/.bash_profile -R headers.properties
>
>   Description
>
> Patch to address FLUME 1096. Please review and share your thoughts.
>
>   Testing
>
> 1. Configured flume agent to publish data in /data/flume/%D
> 2. ran the avroClient to stream some data.
> 3. Things looked ok. Data was available in HDFS.
>
>
>   *Bugs: * https://issues.apache.org/jira/browse/FLUME-1096<https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096>
> Diffs (updated)
>
>    - trunk/bin/flume-ng (1328221)
>    - trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
>    (1328221)
>
> View Diff <https://reviews.apache.org/r/4622/diff/>
>



-- 
Thanks,
- Inder
  Tech Platforms @Inmobi
  Linkedin - http://goo.gl/eR4Ub

Re: Review Request: Patch addressing Issues in using AvroClient with HDFSEventSink for regex based destination paths

Posted by Inder Singh <in...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/
-----------------------------------------------------------

(Updated 2012-04-20 08:41:44.370859)


Review request for Flume, Arvind Prabhakar and Mike Percy.


Changes
-------

Incorporated the following review comments

1. header's come through a config file.
Sample header.properties
streamName=sample-test-stream
timestamp=2329329399

Agent Command Line looks like ->
./flume-ng avro-client --conf ../conf/ -H localhost -p 41414 -F ~/.bash_profile -R headers.properties


Summary
-------

Patch to address FLUME 1096. Please review and share your thoughts.


This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
    https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096


Diffs (updated)
-----

  trunk/bin/flume-ng 1328221 
  trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1328221 

Diff: https://reviews.apache.org/r/4622/diff


Testing
-------

1. Configured flume agent to publish data in /data/flume/%D
2. ran the avroClient to stream some data.
3. Things looked ok. Data was available in HDFS.


Thanks,

Inder

Re: Review Request: Patch addressing Issues in using AvroClient with HDFSEventSink for regex based destination paths

Posted by Inder Singh <in...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/
-----------------------------------------------------------

(Updated 2012-04-09 09:52:21.708833)


Review request for Flume, Arvind Prabhakar and Mike Percy.


Summary
-------

Patch to address FLUME 1096. Please review and share your thoughts.


This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
    https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096


Diffs (updated)
-----

  trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1311132 
  trunk/bin/flume-ng 1311132 

Diff: https://reviews.apache.org/r/4622/diff


Testing
-------

1. Configured flume agent to publish data in /data/flume/%D
2. ran the avroClient to stream some data.
3. Things looked ok. Data was available in HDFS.


Thanks,

Inder

Re: Review Request: Patch addressing Issues in using AvroClient with HDFSEventSink for regex based destination paths

Posted by Inder Singh <in...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4622/
-----------------------------------------------------------

(Updated 2012-04-09 09:49:44.457612)


Review request for Flume, Arvind Prabhakar and Mike Percy.


Changes
-------

Patch to support passing headers in AvroCLIClient


Summary
-------

Patch to address FLUME 1096. Please review and share your thoughts.


This addresses bug https://issues.apache.org/jira/browse/FLUME-1096.
    https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1096


Diffs (updated)
-----

  trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 1311132 

Diff: https://reviews.apache.org/r/4622/diff


Testing
-------

1. Configured flume agent to publish data in /data/flume/%D
2. ran the avroClient to stream some data.
3. Things looked ok. Data was available in HDFS.


Thanks,

Inder