You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Hari Shreedharan <hs...@cloudera.com> on 2013/02/07 04:38:22 UTC
Review Request: FLUME-1898. Thrift Source
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9352/
-----------------------------------------------------------
Review request for Flume.
Description
-------
Thrift Source.
This addresses bug FLUME-1898.
https://issues.apache.org/jira/browse/FLUME-1898
Diffs
-----
flume-ng-core/pom.xml 7db4f23
flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/source/TestThriftSource.java PRE-CREATION
Diff: https://reviews.apache.org/r/9352/diff/
Testing
-------
Added several unit tests.
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1898. Thrift Source
Posted by Hari Shreedharan <hs...@cloudera.com>.
> On Feb. 10, 2013, 9:04 p.m., Brock Noland wrote:
> > Looks great! A few comments below. Also, does this have to be added to the dist pom? I know I have had trouble with things like the JMS source because I didn't add it to the n requisite places.
No. None of these add new modules. All of them are added to the existing modules, so we don't need to add it to the dist pom.
> On Feb. 10, 2013, 9:04 p.m., Brock Noland wrote:
> > flume-ng-core/src/test/java/org/apache/flume/source/TestThriftSource.java, line 64
> > <https://reviews.apache.org/r/9352/diff/4/?file=257024#file257024line64>
> >
> > This is a more reliable mechanism to find a free port:
> >
> > ServerSocket socket = new ServerSocket(0);
> > int port = socket.getLocalPort();
> > socket.close();
Ok, I will add it in a follow-up jira, since I need to update this in TestThriftRpcClient code too, which means I will need to update that patch too (if I update it here, rb will complain that the file does not exist). So I will update both once both these are committed
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9352/#review16398
-----------------------------------------------------------
On Feb. 8, 2013, 9:36 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9352/
> -----------------------------------------------------------
>
> (Updated Feb. 8, 2013, 9:36 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Thrift Source.
>
>
> This addresses bug FLUME-1898.
> https://issues.apache.org/jira/browse/FLUME-1898
>
>
> Diffs
> -----
>
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java 3312b04
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java 058ca1c
> flume-ng-core/pom.xml 7db4f23
> flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/source/TestThriftSource.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/9352/diff/
>
>
> Testing
> -------
>
> Added several unit tests.
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1898. Thrift Source
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9352/#review16398
-----------------------------------------------------------
Looks great! A few comments below. Also, does this have to be added to the dist pom? I know I have had trouble with things like the JMS source because I didn't add it to the n requisite places.
flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java
<https://reviews.apache.org/r/9352/#comment34882>
I don't see a formatting place for this string. Not sur what log4j will do with the second string passed as an arg.
flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java
<https://reviews.apache.org/r/9352/#comment34884>
Do we want to log the exception or exception message here?
flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java
<https://reviews.apache.org/r/9352/#comment34883>
Does log4j accept string formatting? Typically I see {} being used.
flume-ng-core/src/test/java/org/apache/flume/source/TestThriftSource.java
<https://reviews.apache.org/r/9352/#comment34885>
This is a more reliable mechanism to find a free port:
ServerSocket socket = new ServerSocket(0);
int port = socket.getLocalPort();
socket.close();
- Brock Noland
On Feb. 8, 2013, 9:36 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9352/
> -----------------------------------------------------------
>
> (Updated Feb. 8, 2013, 9:36 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Thrift Source.
>
>
> This addresses bug FLUME-1898.
> https://issues.apache.org/jira/browse/FLUME-1898
>
>
> Diffs
> -----
>
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java 3312b04
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java 058ca1c
> flume-ng-core/pom.xml 7db4f23
> flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/source/TestThriftSource.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/9352/diff/
>
>
> Testing
> -------
>
> Added several unit tests.
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1898. Thrift Source
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9352/#review16433
-----------------------------------------------------------
Ship it!
Ship It!
- Brock Noland
On Feb. 10, 2013, 10:50 p.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9352/
> -----------------------------------------------------------
>
> (Updated Feb. 10, 2013, 10:50 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Thrift Source.
>
>
> This addresses bug FLUME-1898.
> https://issues.apache.org/jira/browse/FLUME-1898
>
>
> Diffs
> -----
>
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java 3312b04
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java 058ca1c
> flume-ng-core/pom.xml 7db4f23
> flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/source/TestThriftSource.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/9352/diff/
>
>
> Testing
> -------
>
> Added several unit tests.
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1898. Thrift Source
Posted by Hari Shreedharan <hs...@cloudera.com>.
> On Feb. 11, 2013, 4:15 p.m., Brock Noland wrote:
> > Hi,
> >
> > Patch looks good, I tried running tests and got the following which does not occur on naked trunk:
> >
> > [INFO] Scanning for projects...
> > [ERROR] The build could not read 1 project -> [Help 1]
> > [ERROR]
> > [ERROR] The project org.apache.flume:flume-ng-core:1.4.0-SNAPSHOT (/Users/noland/workspaces/flume-apache-trunk-golden/flume/flume-ng-core/pom.xml) has 1 error
> > [ERROR] 'dependencies.dependency.version' for org.apache.flume:flume-ng-sdk:jar:tests is missing. @ line 140, column 17
> > [ERROR]
> > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
> > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > [ERROR]
> > [ERROR] For more information about the errors and possible solutions, please read the following articles:
> > [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
To fix this, you need to apply the patch for FLUME-1896, then do mvn install and then run this. I added a tests jar to the Flume SDK to use the handlers in the tests there.
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9352/#review16422
-----------------------------------------------------------
On Feb. 10, 2013, 10:50 p.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9352/
> -----------------------------------------------------------
>
> (Updated Feb. 10, 2013, 10:50 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Thrift Source.
>
>
> This addresses bug FLUME-1898.
> https://issues.apache.org/jira/browse/FLUME-1898
>
>
> Diffs
> -----
>
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java 3312b04
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java 058ca1c
> flume-ng-core/pom.xml 7db4f23
> flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/source/TestThriftSource.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/9352/diff/
>
>
> Testing
> -------
>
> Added several unit tests.
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1898. Thrift Source
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9352/#review16422
-----------------------------------------------------------
Hi,
Patch looks good, I tried running tests and got the following which does not occur on naked trunk:
[INFO] Scanning for projects...
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project org.apache.flume:flume-ng-core:1.4.0-SNAPSHOT (/Users/noland/workspaces/flume-apache-trunk-golden/flume/flume-ng-core/pom.xml) has 1 error
[ERROR] 'dependencies.dependency.version' for org.apache.flume:flume-ng-sdk:jar:tests is missing. @ line 140, column 17
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
- Brock Noland
On Feb. 10, 2013, 10:50 p.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9352/
> -----------------------------------------------------------
>
> (Updated Feb. 10, 2013, 10:50 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Thrift Source.
>
>
> This addresses bug FLUME-1898.
> https://issues.apache.org/jira/browse/FLUME-1898
>
>
> Diffs
> -----
>
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java 3312b04
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java 058ca1c
> flume-ng-core/pom.xml 7db4f23
> flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/source/TestThriftSource.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/9352/diff/
>
>
> Testing
> -------
>
> Added several unit tests.
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1898. Thrift Source
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9352/
-----------------------------------------------------------
(Updated Feb. 10, 2013, 10:50 p.m.)
Review request for Flume.
Changes
-------
Brock's feedback updates.
Description
-------
Thrift Source.
This addresses bug FLUME-1898.
https://issues.apache.org/jira/browse/FLUME-1898
Diffs (updated)
-----
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java 3312b04
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java 058ca1c
flume-ng-core/pom.xml 7db4f23
flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/source/TestThriftSource.java PRE-CREATION
Diff: https://reviews.apache.org/r/9352/diff/
Testing
-------
Added several unit tests.
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1898. Thrift Source
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9352/
-----------------------------------------------------------
(Updated Feb. 8, 2013, 9:36 a.m.)
Review request for Flume.
Changes
-------
Adding some pom changes to this patch, because RB won't let me add this to the Avro Sink patch, because the flume-ng-core/pom.xml file was modified in this patch.
Description
-------
Thrift Source.
This addresses bug FLUME-1898.
https://issues.apache.org/jira/browse/FLUME-1898
Diffs (updated)
-----
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java 3312b04
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java 058ca1c
flume-ng-core/pom.xml 7db4f23
flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/source/TestThriftSource.java PRE-CREATION
Diff: https://reviews.apache.org/r/9352/diff/
Testing
-------
Added several unit tests.
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1898. Thrift Source
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9352/
-----------------------------------------------------------
(Updated Feb. 7, 2013, 10:47 p.m.)
Review request for Flume.
Changes
-------
Add to SourceType enum
Description
-------
Thrift Source.
This addresses bug FLUME-1898.
https://issues.apache.org/jira/browse/FLUME-1898
Diffs (updated)
-----
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java 3312b04
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java 058ca1c
flume-ng-core/pom.xml 7db4f23
flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/source/TestThriftSource.java PRE-CREATION
Diff: https://reviews.apache.org/r/9352/diff/
Testing
-------
Added several unit tests.
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1898. Thrift Source
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9352/
-----------------------------------------------------------
(Updated Feb. 7, 2013, 10:03 p.m.)
Review request for Flume.
Changes
-------
Update based on Brock's feedback
Description
-------
Thrift Source.
This addresses bug FLUME-1898.
https://issues.apache.org/jira/browse/FLUME-1898
Diffs (updated)
-----
flume-ng-core/pom.xml 7db4f23
flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/source/TestThriftSource.java PRE-CREATION
Diff: https://reviews.apache.org/r/9352/diff/
Testing
-------
Added several unit tests.
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1898. Thrift Source
Posted by Hari Shreedharan <hs...@cloudera.com>.
> On Feb. 7, 2013, 3:23 p.m., Brock Noland wrote:
> > flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java, line 110
> > <https://reviews.apache.org/r/9352/diff/1/?file=256554#file256554line110>
> >
> > If this ex is thrown, I assume serverTransport will be null. Will this be an issue?
Good catch, I missed throwing the exception after wrapping it in a FlumeException. Else the server.serve() or creating the server would have thrown.
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9352/#review16273
-----------------------------------------------------------
On Feb. 7, 2013, 10:03 p.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9352/
> -----------------------------------------------------------
>
> (Updated Feb. 7, 2013, 10:03 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Thrift Source.
>
>
> This addresses bug FLUME-1898.
> https://issues.apache.org/jira/browse/FLUME-1898
>
>
> Diffs
> -----
>
> flume-ng-core/pom.xml 7db4f23
> flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/source/TestThriftSource.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/9352/diff/
>
>
> Testing
> -------
>
> Added several unit tests.
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1898. Thrift Source
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9352/#review16273
-----------------------------------------------------------
Hari, looks like a nice simple patch! A few comments below.
flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java
<https://reviews.apache.org/r/9352/#comment34728>
If not specified this will throw an NPE. Could we have a better error message?
flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java
<https://reviews.apache.org/r/9352/#comment34727>
Could we have a better error message here?
flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java
<https://reviews.apache.org/r/9352/#comment34725>
If this ex is thrown, I assume serverTransport will be null. Will this be an issue?
flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java
<https://reviews.apache.org/r/9352/#comment34726>
nit: spacing
- Brock Noland
On Feb. 7, 2013, 3:38 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9352/
> -----------------------------------------------------------
>
> (Updated Feb. 7, 2013, 3:38 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Thrift Source.
>
>
> This addresses bug FLUME-1898.
> https://issues.apache.org/jira/browse/FLUME-1898
>
>
> Diffs
> -----
>
> flume-ng-core/pom.xml 7db4f23
> flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/source/TestThriftSource.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/9352/diff/
>
>
> Testing
> -------
>
> Added several unit tests.
>
>
> Thanks,
>
> Hari Shreedharan
>
>