You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Mike Percy (Created) (JIRA)" <ji...@apache.org> on 2012/02/23 04:39:53 UTC
[jira] [Created] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Factor Flume Avro RPC interfaces out into separate Client SDK
-------------------------------------------------------------
Key: FLUME-989
URL: https://issues.apache.org/jira/browse/FLUME-989
Project: Flume
Issue Type: Sub-task
Affects Versions: v1.0.0
Reporter: Mike Percy
Fix For: v1.1.0
Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217621#comment-13217621 ]
jiraposter@reviews.apache.org commented on FLUME-989:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4047/#review5361
-----------------------------------------------------------
Thanks for the patch Mike. The changes look good. Some suggestions follow:
1. Please use 80 char limit on line lengths and wrap the lines where necessary. Also the general formatting/style convention is that we should use spaces only, two-spaces per tab and no trailing whitespaces in the sources. Any trailing whitespaces are highlighted with red in the reviewboard.
2. An event builder in the org.apache.flume package that can be used to create events with a single string or object will be a great complementing addition.
3. Now that the simple to use client API is there, we should refactor the AvroCLIClient and Log4jAppender to use it instead of directly going via the low-level API.
flume-ng-sdk/src/main/java/org/apache/flume/api/client/AvroClientBuilder.java
<https://reviews.apache.org/r/4047/#comment11668>
Adding a check on batch size should be good too.
- Arvind
On 2012-02-24 22:47:13, Mike Percy wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4047/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-24 22:47:13)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Seeking early feedback on some additional APIs to make integrating with Flume 1.x easier.
bq.
bq. Added the following APIs:
bq. - AvroClient: Friendly Java interface around the Avro API
bq. - AvroClientBuilder: Builder class to allow easy extension of AvroClient capabilities in the future (i.e. SSL)
bq. - DefaultAvroClient: Implementation of the AvroClient interface
bq.
bq. Created this stuff in a flume-ng-sdk Maven submodule and moved the Event interface to that submodule. flume-ng-core depends on flume-ng-sdk.
bq.
bq. I also modified AvroSink to use the AvroClient API instead of the bare AvroSourceProtocol API directly.
bq.
bq.
bq. This addresses bug FLUME-989.
bq. https://issues.apache.org/jira/browse/FLUME-989
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-core/pom.xml d753fa1
bq. flume-ng-core/src/main/avro/flume.avdl 40da3ef
bq. flume-ng-core/src/main/java/org/apache/flume/Event.java a017705
bq. flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
bq. flume-ng-sdk/pom.xml PRE-CREATION
bq. flume-ng-sdk/src/main/avro/flume.avdl PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/Event.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/client/AvroClient.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/client/AvroClientBuilder.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/client/DefaultAvroClient.java PRE-CREATION
bq. pom.xml d785762
bq.
bq. Diff: https://reviews.apache.org/r/4047/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Mike
bq.
bq.
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217473#comment-13217473 ]
jiraposter@reviews.apache.org commented on FLUME-989:
-----------------------------------------------------
bq. On 2012-02-27 19:50:55, Hari Shreedharan wrote:
bq. >
Looks good, though I was wondering if the DefaultAvroClient class is thread safe.
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4047/#review5357
-----------------------------------------------------------
On 2012-02-24 22:47:13, Mike Percy wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4047/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-24 22:47:13)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Seeking early feedback on some additional APIs to make integrating with Flume 1.x easier.
bq.
bq. Added the following APIs:
bq. - AvroClient: Friendly Java interface around the Avro API
bq. - AvroClientBuilder: Builder class to allow easy extension of AvroClient capabilities in the future (i.e. SSL)
bq. - DefaultAvroClient: Implementation of the AvroClient interface
bq.
bq. Created this stuff in a flume-ng-sdk Maven submodule and moved the Event interface to that submodule. flume-ng-core depends on flume-ng-sdk.
bq.
bq. I also modified AvroSink to use the AvroClient API instead of the bare AvroSourceProtocol API directly.
bq.
bq.
bq. This addresses bug FLUME-989.
bq. https://issues.apache.org/jira/browse/FLUME-989
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-core/pom.xml d753fa1
bq. flume-ng-core/src/main/avro/flume.avdl 40da3ef
bq. flume-ng-core/src/main/java/org/apache/flume/Event.java a017705
bq. flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
bq. flume-ng-sdk/pom.xml PRE-CREATION
bq. flume-ng-sdk/src/main/avro/flume.avdl PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/Event.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/client/AvroClient.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/client/AvroClientBuilder.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/client/DefaultAvroClient.java PRE-CREATION
bq. pom.xml d785762
bq.
bq. Diff: https://reviews.apache.org/r/4047/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Mike
bq.
bq.
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217469#comment-13217469 ]
jiraposter@reviews.apache.org commented on FLUME-989:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4047/#review5357
-----------------------------------------------------------
flume-ng-sdk/src/main/java/org/apache/flume/api/client/AvroClientBuilder.java
<https://reviews.apache.org/r/4047/#comment11660>
Maybe you can provide a version of build which simply takes the 2 arguments directly as well. You can simply set the values in the object and then call this version of build().
flume-ng-sdk/src/main/java/org/apache/flume/api/client/DefaultAvroClient.java
<https://reviews.apache.org/r/4047/#comment11661>
I think close, append and appendBatch should be synchronized. If something like this happens:
if(!isConnected) runs in one thread and then some other thread closes it, the append call(in the first thread) is going to try to connect to a transceiver that is closed, even though the append call was made before the close. Similar case is true for appendBatch, also I think.
- Hari
On 2012-02-24 22:47:13, Mike Percy wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4047/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-24 22:47:13)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Seeking early feedback on some additional APIs to make integrating with Flume 1.x easier.
bq.
bq. Added the following APIs:
bq. - AvroClient: Friendly Java interface around the Avro API
bq. - AvroClientBuilder: Builder class to allow easy extension of AvroClient capabilities in the future (i.e. SSL)
bq. - DefaultAvroClient: Implementation of the AvroClient interface
bq.
bq. Created this stuff in a flume-ng-sdk Maven submodule and moved the Event interface to that submodule. flume-ng-core depends on flume-ng-sdk.
bq.
bq. I also modified AvroSink to use the AvroClient API instead of the bare AvroSourceProtocol API directly.
bq.
bq.
bq. This addresses bug FLUME-989.
bq. https://issues.apache.org/jira/browse/FLUME-989
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-core/pom.xml d753fa1
bq. flume-ng-core/src/main/avro/flume.avdl 40da3ef
bq. flume-ng-core/src/main/java/org/apache/flume/Event.java a017705
bq. flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
bq. flume-ng-sdk/pom.xml PRE-CREATION
bq. flume-ng-sdk/src/main/avro/flume.avdl PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/Event.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/client/AvroClient.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/client/AvroClientBuilder.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/client/DefaultAvroClient.java PRE-CREATION
bq. pom.xml d785762
bq.
bq. Diff: https://reviews.apache.org/r/4047/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Mike
bq.
bq.
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225141#comment-13225141 ]
jiraposter@reviews.apache.org commented on FLUME-989:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4047/
-----------------------------------------------------------
(Updated 2012-03-08 11:14:12.062221)
Review request for Flume.
Changes
-------
I have attached a new diff containing an updated Client RPC API. Hoping for feedback on the approach.
All current unit tests pass. I will also add additional unit tests to specifically exercise this API.
Summary
-------
Seeking early feedback on some additional APIs to make integrating with Flume 1.x easier.
Added the following APIs:
- AvroClient: Friendly Java interface around the Avro API
- AvroClientBuilder: Builder class to allow easy extension of AvroClient capabilities in the future (i.e. SSL)
- DefaultAvroClient: Implementation of the AvroClient interface
Created this stuff in a flume-ng-sdk Maven submodule and moved the Event interface to that submodule. flume-ng-core depends on flume-ng-sdk.
I also modified AvroSink to use the AvroClient API instead of the bare AvroSourceProtocol API directly.
This addresses bug FLUME-989.
https://issues.apache.org/jira/browse/FLUME-989
Diffs (updated)
-----
flume-ng-clients/flume-ng-log4jappender/pom.xml 9dd31bd
flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java 97f2b9e
flume-ng-core/pom.xml fe6ce0b
flume-ng-core/src/main/avro/flume.avdl 40da3ef
flume-ng-core/src/main/java/org/apache/flume/Event.java a017705
flume-ng-core/src/main/java/org/apache/flume/FlumeException.java eab5b3d
flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 195ba79
flume-ng-core/src/main/java/org/apache/flume/event/EventBuilder.java 5d8c3b3
flume-ng-core/src/main/java/org/apache/flume/event/SimpleEvent.java e0c3b45
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-sdk/pom.xml PRE-CREATION
flume-ng-sdk/src/main/avro/flume.avdl PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/Event.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/FlumeRemoteException.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/FlumeTimeoutException.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/client/NettyAvroRpcClient.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcCallback.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClient.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClientBuilder.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcResponse.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/event/EventBuilder.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/event/SimpleEvent.java PRE-CREATION
pom.xml d785762
Diff: https://reviews.apache.org/r/4047/diff
Testing
-------
Thanks,
Mike
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "Mike Percy (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Percy reassigned FLUME-989:
--------------------------------
Assignee: Mike Percy
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226015#comment-13226015 ]
jiraposter@reviews.apache.org commented on FLUME-989:
-----------------------------------------------------
bq. On 2012-03-08 18:29:17, Arvind Prabhakar wrote:
bq. > flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcCallback.java, line 18
bq. > <https://reviews.apache.org/r/4047/diff/2/?file=89096#file89096line18>
bq. >
bq. > Unused interface, should be removed.
Done
bq. On 2012-03-08 18:29:17, Arvind Prabhakar wrote:
bq. > flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClient.java, line 28
bq. > <https://reviews.apache.org/r/4047/diff/2/?file=89097#file89097line28>
bq. >
bq. > I suggest removing the FlumeRemoteException and FlumeTimeoutException in favor of EventDeliveryException as that is what the client is concerned with from user's perspective.
bq. >
bq. > Exposing these finer grain exceptions allow the creation of more sophisticated clients which runs contrary to the goal of keeping the client as simple to code as possible. Once we have concrete use-case from the field where exposing this semantic is necessary, we can easily introduce it at that time.
bq. >
bq. > Similarly we should remove the RpcResponse altogether. For blocking calls, the best way to communicate failure is via the exception and any more state insight that RpcResponse gives will add to the complexity of the client layer.
Done. Note that EventDeliveryException is checked and we should consider making it inherit from FlumeException for consistency. But I didn't try to make that change here since it would not be backwards compatible.
bq. On 2012-03-08 18:29:17, Arvind Prabhakar wrote:
bq. > flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClient.java, line 52
bq. > <https://reviews.apache.org/r/4047/diff/2/?file=89097#file89097line52>
bq. >
bq. > This should be removed and the functionality should be managed within the client implementation via configuration.
I simply removed the functionality from the interface and the default timeouts are used only for now.
bq. On 2012-03-08 18:29:17, Arvind Prabhakar wrote:
bq. > flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClient.java, line 80
bq. > <https://reviews.apache.org/r/4047/diff/2/?file=89097#file89097line80>
bq. >
bq. > Same here as the previous comment.
bq. >
Done.
bq. On 2012-03-08 18:29:17, Arvind Prabhakar wrote:
bq. > flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClient.java, line 85
bq. > <https://reviews.apache.org/r/4047/diff/2/?file=89097#file89097line85>
bq. >
bq. > no such method #appendBatch(List, RpcCallback)
Thanks, fixed!
bq. On 2012-03-08 18:29:17, Arvind Prabhakar wrote:
bq. > flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClient.java, line 106
bq. > <https://reviews.apache.org/r/4047/diff/2/?file=89097#file89097line106>
bq. >
bq. > nit: The regular idiom is to check for positive and not negative. So it would make sense for this method to be isActive() instead.
I just removed it, as mentioned above.
bq. On 2012-03-08 18:29:17, Arvind Prabhakar wrote:
bq. > flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClientBuilder.java, line 90
bq. > <https://reviews.apache.org/r/4047/diff/2/?file=89098#file89098line90>
bq. >
bq. > This should not be exposed but read via configuration.
Hid this functionality for now.
bq. On 2012-03-08 18:29:17, Arvind Prabhakar wrote:
bq. > flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClientBuilder.java, lines 121-123
bq. > <https://reviews.apache.org/r/4047/diff/2/?file=89098#file89098line121>
bq. >
bq. > These checks seem redundant since the builder will check them again anyway.
Done.
On 2012-03-08 18:29:17, Mike Percy wrote:
bq. > Finally, please include some direct tests that exercise this API.
Added a bunch of tests that really put the API through its paces.
- Mike
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4047/#review5734
-----------------------------------------------------------
On 2012-03-09 11:14:13, Mike Percy wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4047/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-09 11:14:13)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Seeking early feedback on some additional APIs to make integrating with Flume 1.x easier.
bq.
bq. Added the following APIs:
bq. - AvroClient: Friendly Java interface around the Avro API
bq. - AvroClientBuilder: Builder class to allow easy extension of AvroClient capabilities in the future (i.e. SSL)
bq. - DefaultAvroClient: Implementation of the AvroClient interface
bq.
bq. Created this stuff in a flume-ng-sdk Maven submodule and moved the Event interface to that submodule. flume-ng-core depends on flume-ng-sdk.
bq.
bq. I also modified AvroSink to use the AvroClient API instead of the bare AvroSourceProtocol API directly.
bq.
bq.
bq. This addresses bug FLUME-989.
bq. https://issues.apache.org/jira/browse/FLUME-989
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-clients/flume-ng-log4jappender/pom.xml 9dd31bd
bq. flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java 97f2b9e
bq. flume-ng-core/pom.xml fe6ce0b
bq. flume-ng-core/src/main/avro/flume.avdl 40da3ef
bq. flume-ng-core/src/main/java/org/apache/flume/Event.java 5278fc0
bq. flume-ng-core/src/main/java/org/apache/flume/EventDeliveryException.java 1413223
bq. flume-ng-core/src/main/java/org/apache/flume/FlumeException.java eab5b3d
bq. flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 195ba79
bq. flume-ng-core/src/main/java/org/apache/flume/event/EventBuilder.java 5d8c3b3
bq. flume-ng-core/src/main/java/org/apache/flume/event/SimpleEvent.java e0c3b45
bq. flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
bq. flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
bq. flume-ng-core/src/test/java/org/apache/flume/util/TestEventBuilder.java 7930607
bq. flume-ng-sdk/pom.xml PRE-CREATION
bq. flume-ng-sdk/src/main/avro/flume.avdl PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/Event.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/EventDeliveryException.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/NettyAvroRpcClient.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClient.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientFactory.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/event/EventBuilder.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/event/SimpleEvent.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/api/NettyAvroRpcTestHelpers.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/api/TestRpcClientFactory.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/event/TestEventBuilder.java PRE-CREATION
bq. pom.xml d785762
bq.
bq. Diff: https://reviews.apache.org/r/4047/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Mike
bq.
bq.
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "Mike Percy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Percy updated FLUME-989:
-----------------------------
Status: Patch Available (was: Open)
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
> Attachments: FLUME-989-21.patch
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216031#comment-13216031 ]
jiraposter@reviews.apache.org commented on FLUME-989:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4047/
-----------------------------------------------------------
Review request for Flume.
Summary
-------
Seeking early feedback on some additional APIs to make integrating with Flume 1.x easier.
Added the following APIs:
- AvroClient: Friendly Java interface around the Avro API
- AvroClientBuilder: Builder class to allow easy extension of AvroClient capabilities in the future (i.e. SSL)
- DefaultAvroClient: Implementation of the AvroClient interface
Created this stuff in a flume-ng-sdk Maven submodule and moved the Event interface to that submodule. flume-ng-core depends on flume-ng-sdk.
I also modified AvroSink to use the AvroClient API instead of the bare AvroSourceProtocol API directly.
This addresses bug FLUME-989.
https://issues.apache.org/jira/browse/FLUME-989
Diffs
-----
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/avro/flume.avdl 40da3ef
flume-ng-core/src/main/java/org/apache/flume/Event.java a017705
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
flume-ng-sdk/pom.xml PRE-CREATION
flume-ng-sdk/src/main/avro/flume.avdl PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/Event.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/client/AvroClient.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/client/AvroClientBuilder.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/client/DefaultAvroClient.java PRE-CREATION
pom.xml d785762
Diff: https://reviews.apache.org/r/4047/diff
Testing
-------
Thanks,
Mike
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228162#comment-13228162 ]
Hudson commented on FLUME-989:
------------------------------
Integrated in flume-trunk #120 (See [https://builds.apache.org/job/flume-trunk/120/])
FLUME-989. Factor Flume Avro RPC interfaces out in separate Client SDK.
(Mike Percy via Arvind Prabhakar) (Revision 1299958)
Result = FAILURE
arvind : http://svn.apache.org/viewvc/?view=rev&rev=1299958
Files :
* /incubator/flume/trunk/flume-ng-channels/flume-file-channel/pom.xml
* /incubator/flume/trunk/flume-ng-channels/flume-jdbc-channel/pom.xml
* /incubator/flume/trunk/flume-ng-clients/flume-ng-log4jappender/pom.xml
* /incubator/flume/trunk/flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java
* /incubator/flume/trunk/flume-ng-core/pom.xml
* /incubator/flume/trunk/flume-ng-core/src/main/avro/flume.avdl
* /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/Event.java
* /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/EventDeliveryException.java
* /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/FlumeException.java
* /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
* /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/event/EventBuilder.java
* /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/event/SimpleEvent.java
* /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java
* /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java
* /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java
* /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java
* /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/util/TestEventBuilder.java
* /incubator/flume/trunk/flume-ng-legacy-sources/flume-avro-source/pom.xml
* /incubator/flume/trunk/flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java
* /incubator/flume/trunk/flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java
* /incubator/flume/trunk/flume-ng-legacy-sources/flume-thrift-source/pom.xml
* /incubator/flume/trunk/flume-ng-node/pom.xml
* /incubator/flume/trunk/flume-ng-sdk
* /incubator/flume/trunk/flume-ng-sdk/pom.xml
* /incubator/flume/trunk/flume-ng-sdk/src
* /incubator/flume/trunk/flume-ng-sdk/src/main
* /incubator/flume/trunk/flume-ng-sdk/src/main/avro
* /incubator/flume/trunk/flume-ng-sdk/src/main/avro/flume.avdl
* /incubator/flume/trunk/flume-ng-sdk/src/main/java
* /incubator/flume/trunk/flume-ng-sdk/src/main/java/org
* /incubator/flume/trunk/flume-ng-sdk/src/main/java/org/apache
* /incubator/flume/trunk/flume-ng-sdk/src/main/java/org/apache/flume
* /incubator/flume/trunk/flume-ng-sdk/src/main/java/org/apache/flume/Event.java
* /incubator/flume/trunk/flume-ng-sdk/src/main/java/org/apache/flume/EventDeliveryException.java
* /incubator/flume/trunk/flume-ng-sdk/src/main/java/org/apache/flume/FlumeException.java
* /incubator/flume/trunk/flume-ng-sdk/src/main/java/org/apache/flume/api
* /incubator/flume/trunk/flume-ng-sdk/src/main/java/org/apache/flume/api/NettyAvroRpcClient.java
* /incubator/flume/trunk/flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClient.java
* /incubator/flume/trunk/flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientFactory.java
* /incubator/flume/trunk/flume-ng-sdk/src/main/java/org/apache/flume/event
* /incubator/flume/trunk/flume-ng-sdk/src/main/java/org/apache/flume/event/EventBuilder.java
* /incubator/flume/trunk/flume-ng-sdk/src/main/java/org/apache/flume/event/SimpleEvent.java
* /incubator/flume/trunk/flume-ng-sdk/src/test
* /incubator/flume/trunk/flume-ng-sdk/src/test/java
* /incubator/flume/trunk/flume-ng-sdk/src/test/java/org
* /incubator/flume/trunk/flume-ng-sdk/src/test/java/org/apache
* /incubator/flume/trunk/flume-ng-sdk/src/test/java/org/apache/flume
* /incubator/flume/trunk/flume-ng-sdk/src/test/java/org/apache/flume/api
* /incubator/flume/trunk/flume-ng-sdk/src/test/java/org/apache/flume/api/RpcTestUtils.java
* /incubator/flume/trunk/flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java
* /incubator/flume/trunk/flume-ng-sdk/src/test/java/org/apache/flume/api/TestRpcClientFactory.java
* /incubator/flume/trunk/flume-ng-sdk/src/test/java/org/apache/flume/event
* /incubator/flume/trunk/flume-ng-sdk/src/test/java/org/apache/flume/event/TestEventBuilder.java
* /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/pom.xml
* /incubator/flume/trunk/flume-ng-sinks/flume-irc-sink/pom.xml
* /incubator/flume/trunk/pom.xml
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
> Attachments: FLUME-989-21.patch
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225366#comment-13225366 ]
jiraposter@reviews.apache.org commented on FLUME-989:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4047/#review5734
-----------------------------------------------------------
Thanks for the patch Mike. The changes are coming together nicely. I do have some feedback that follows.
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcCallback.java
<https://reviews.apache.org/r/4047/#comment12482>
Unused interface, should be removed.
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClient.java
<https://reviews.apache.org/r/4047/#comment12473>
I suggest removing the FlumeRemoteException and FlumeTimeoutException in favor of EventDeliveryException as that is what the client is concerned with from user's perspective.
Exposing these finer grain exceptions allow the creation of more sophisticated clients which runs contrary to the goal of keeping the client as simple to code as possible. Once we have concrete use-case from the field where exposing this semantic is necessary, we can easily introduce it at that time.
Similarly we should remove the RpcResponse altogether. For blocking calls, the best way to communicate failure is via the exception and any more state insight that RpcResponse gives will add to the complexity of the client layer.
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClient.java
<https://reviews.apache.org/r/4047/#comment12474>
This should be removed and the functionality should be managed within the client implementation via configuration.
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClient.java
<https://reviews.apache.org/r/4047/#comment12475>
Same here as the previous comment.
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClient.java
<https://reviews.apache.org/r/4047/#comment12481>
no such method #appendBatch(List, RpcCallback)
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClient.java
<https://reviews.apache.org/r/4047/#comment12471>
nit: The regular idiom is to check for positive and not negative. So it would make sense for this method to be isActive() instead.
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClientBuilder.java
<https://reviews.apache.org/r/4047/#comment12480>
This should not be exposed but read via configuration.
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClientBuilder.java
<https://reviews.apache.org/r/4047/#comment12476>
These checks seem redundant since the builder will check them again anyway.
Finally, please include some direct tests that exercise this API.
- Arvind
On 2012-03-08 11:14:53, Mike Percy wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4047/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-08 11:14:53)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Seeking early feedback on some additional APIs to make integrating with Flume 1.x easier.
bq.
bq. Added the following APIs:
bq. - AvroClient: Friendly Java interface around the Avro API
bq. - AvroClientBuilder: Builder class to allow easy extension of AvroClient capabilities in the future (i.e. SSL)
bq. - DefaultAvroClient: Implementation of the AvroClient interface
bq.
bq. Created this stuff in a flume-ng-sdk Maven submodule and moved the Event interface to that submodule. flume-ng-core depends on flume-ng-sdk.
bq.
bq. I also modified AvroSink to use the AvroClient API instead of the bare AvroSourceProtocol API directly.
bq.
bq.
bq. This addresses bug FLUME-989.
bq. https://issues.apache.org/jira/browse/FLUME-989
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-clients/flume-ng-log4jappender/pom.xml 9dd31bd
bq. flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java 97f2b9e
bq. flume-ng-core/pom.xml fe6ce0b
bq. flume-ng-core/src/main/avro/flume.avdl 40da3ef
bq. flume-ng-core/src/main/java/org/apache/flume/Event.java a017705
bq. flume-ng-core/src/main/java/org/apache/flume/FlumeException.java eab5b3d
bq. flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 195ba79
bq. flume-ng-core/src/main/java/org/apache/flume/event/EventBuilder.java 5d8c3b3
bq. flume-ng-core/src/main/java/org/apache/flume/event/SimpleEvent.java e0c3b45
bq. flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
bq. flume-ng-sdk/pom.xml PRE-CREATION
bq. flume-ng-sdk/src/main/avro/flume.avdl PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/Event.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/FlumeRemoteException.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/FlumeTimeoutException.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/client/NettyAvroRpcClient.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcCallback.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClient.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClientBuilder.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcResponse.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/event/EventBuilder.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/event/SimpleEvent.java PRE-CREATION
bq. pom.xml d785762
bq.
bq. Diff: https://reviews.apache.org/r/4047/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Mike
bq.
bq.
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "Arvind Prabhakar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arvind Prabhakar updated FLUME-989:
-----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch committed. Thanks Mike!
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
> Attachments: FLUME-989-21.patch
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226014#comment-13226014 ]
jiraposter@reviews.apache.org commented on FLUME-989:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4047/
-----------------------------------------------------------
(Updated 2012-03-09 11:14:13.656079)
Review request for Flume.
Changes
-------
I applied all of the feedback as well as went a bit farther. The semantics of isAlive() were kind of weird so I just removed it completely and now if you get an EventDeliveryException you must close the client and construct a new one. It's a bit heavy but it's a contract we can build on as we evolve the system and APIs.
I also found an implementation bug in AvroSink where it will never throw an EventDeliveryException. So I fixed that to be consistent with the behavior we have been discussing w.r.t. Javadocs lately. Now it only uses BACKOFF for an empty feeder channel.
I changed the builder to be an inner class and I left the factory methods as a factory. I think that gives us both flexibility in the future w/ the builder and simplicity for now w/ the factory.
I wrote a whole bunch of unit tests as well. They uncovered some issues which I have addressed.
This patch applies to the latest trunk.
Summary
-------
Seeking early feedback on some additional APIs to make integrating with Flume 1.x easier.
Added the following APIs:
- AvroClient: Friendly Java interface around the Avro API
- AvroClientBuilder: Builder class to allow easy extension of AvroClient capabilities in the future (i.e. SSL)
- DefaultAvroClient: Implementation of the AvroClient interface
Created this stuff in a flume-ng-sdk Maven submodule and moved the Event interface to that submodule. flume-ng-core depends on flume-ng-sdk.
I also modified AvroSink to use the AvroClient API instead of the bare AvroSourceProtocol API directly.
This addresses bug FLUME-989.
https://issues.apache.org/jira/browse/FLUME-989
Diffs (updated)
-----
flume-ng-clients/flume-ng-log4jappender/pom.xml 9dd31bd
flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java 97f2b9e
flume-ng-core/pom.xml fe6ce0b
flume-ng-core/src/main/avro/flume.avdl 40da3ef
flume-ng-core/src/main/java/org/apache/flume/Event.java 5278fc0
flume-ng-core/src/main/java/org/apache/flume/EventDeliveryException.java 1413223
flume-ng-core/src/main/java/org/apache/flume/FlumeException.java eab5b3d
flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 195ba79
flume-ng-core/src/main/java/org/apache/flume/event/EventBuilder.java 5d8c3b3
flume-ng-core/src/main/java/org/apache/flume/event/SimpleEvent.java e0c3b45
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/util/TestEventBuilder.java 7930607
flume-ng-sdk/pom.xml PRE-CREATION
flume-ng-sdk/src/main/avro/flume.avdl PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/Event.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/EventDeliveryException.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/NettyAvroRpcClient.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClient.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientFactory.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/event/EventBuilder.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/event/SimpleEvent.java PRE-CREATION
flume-ng-sdk/src/test/java/org/apache/flume/api/NettyAvroRpcTestHelpers.java PRE-CREATION
flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java PRE-CREATION
flume-ng-sdk/src/test/java/org/apache/flume/api/TestRpcClientFactory.java PRE-CREATION
flume-ng-sdk/src/test/java/org/apache/flume/event/TestEventBuilder.java PRE-CREATION
pom.xml d785762
Diff: https://reviews.apache.org/r/4047/diff
Testing
-------
Thanks,
Mike
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226370#comment-13226370 ]
jiraposter@reviews.apache.org commented on FLUME-989:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4047/#review5798
-----------------------------------------------------------
After sleeping on it, I am going to put isAlive() back. I don't like the semantics of having to catch exceptions and save state for the next call. So please wait for me to update the patch before reviewing.
- Mike
On 2012-03-09 11:14:13, Mike Percy wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4047/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-09 11:14:13)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Seeking early feedback on some additional APIs to make integrating with Flume 1.x easier.
bq.
bq. Added the following APIs:
bq. - AvroClient: Friendly Java interface around the Avro API
bq. - AvroClientBuilder: Builder class to allow easy extension of AvroClient capabilities in the future (i.e. SSL)
bq. - DefaultAvroClient: Implementation of the AvroClient interface
bq.
bq. Created this stuff in a flume-ng-sdk Maven submodule and moved the Event interface to that submodule. flume-ng-core depends on flume-ng-sdk.
bq.
bq. I also modified AvroSink to use the AvroClient API instead of the bare AvroSourceProtocol API directly.
bq.
bq.
bq. This addresses bug FLUME-989.
bq. https://issues.apache.org/jira/browse/FLUME-989
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-clients/flume-ng-log4jappender/pom.xml 9dd31bd
bq. flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java 97f2b9e
bq. flume-ng-core/pom.xml fe6ce0b
bq. flume-ng-core/src/main/avro/flume.avdl 40da3ef
bq. flume-ng-core/src/main/java/org/apache/flume/Event.java 5278fc0
bq. flume-ng-core/src/main/java/org/apache/flume/EventDeliveryException.java 1413223
bq. flume-ng-core/src/main/java/org/apache/flume/FlumeException.java eab5b3d
bq. flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 195ba79
bq. flume-ng-core/src/main/java/org/apache/flume/event/EventBuilder.java 5d8c3b3
bq. flume-ng-core/src/main/java/org/apache/flume/event/SimpleEvent.java e0c3b45
bq. flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
bq. flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
bq. flume-ng-core/src/test/java/org/apache/flume/util/TestEventBuilder.java 7930607
bq. flume-ng-sdk/pom.xml PRE-CREATION
bq. flume-ng-sdk/src/main/avro/flume.avdl PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/Event.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/EventDeliveryException.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/NettyAvroRpcClient.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClient.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientFactory.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/event/EventBuilder.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/event/SimpleEvent.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/api/NettyAvroRpcTestHelpers.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/api/TestRpcClientFactory.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/event/TestEventBuilder.java PRE-CREATION
bq. pom.xml d785762
bq.
bq. Diff: https://reviews.apache.org/r/4047/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Mike
bq.
bq.
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "Mike Percy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Percy updated FLUME-989:
-----------------------------
Attachment: FLUME-989-21.patch
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
> Attachments: FLUME-989-21.patch
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228115#comment-13228115 ]
jiraposter@reviews.apache.org commented on FLUME-989:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4047/#review5866
-----------------------------------------------------------
Ship it!
+1. Changes look good Mike. Please attach the patch to the Jira.
- Arvind
On 2012-03-12 23:33:32, Mike Percy wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4047/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-12 23:33:32)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Seeking early feedback on some additional APIs to make integrating with Flume 1.x easier.
bq.
bq. Added the following APIs:
bq. - AvroClient: Friendly Java interface around the Avro API
bq. - AvroClientBuilder: Builder class to allow easy extension of AvroClient capabilities in the future (i.e. SSL)
bq. - DefaultAvroClient: Implementation of the AvroClient interface
bq.
bq. Created this stuff in a flume-ng-sdk Maven submodule and moved the Event interface to that submodule. flume-ng-core depends on flume-ng-sdk.
bq.
bq. I also modified AvroSink to use the AvroClient API instead of the bare AvroSourceProtocol API directly.
bq.
bq.
bq. This addresses bug FLUME-989.
bq. https://issues.apache.org/jira/browse/FLUME-989
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-channels/flume-file-channel/pom.xml ee2d20f
bq. flume-ng-channels/flume-jdbc-channel/pom.xml 51a7694
bq. flume-ng-clients/flume-ng-log4jappender/pom.xml 9dd31bd
bq. flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java 97f2b9e
bq. flume-ng-core/pom.xml fe6ce0b
bq. flume-ng-core/src/main/avro/flume.avdl 40da3ef
bq. flume-ng-core/src/main/java/org/apache/flume/Event.java 5278fc0
bq. flume-ng-core/src/main/java/org/apache/flume/EventDeliveryException.java 1413223
bq. flume-ng-core/src/main/java/org/apache/flume/FlumeException.java eab5b3d
bq. flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 195ba79
bq. flume-ng-core/src/main/java/org/apache/flume/event/EventBuilder.java 5d8c3b3
bq. flume-ng-core/src/main/java/org/apache/flume/event/SimpleEvent.java e0c3b45
bq. flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
bq. flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
bq. flume-ng-core/src/test/java/org/apache/flume/util/TestEventBuilder.java 7930607
bq. flume-ng-legacy-sources/flume-avro-source/pom.xml 2d53403
bq. flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
bq. flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
bq. flume-ng-legacy-sources/flume-thrift-source/pom.xml 8256f0f
bq. flume-ng-node/pom.xml b9b062e
bq. flume-ng-sdk/pom.xml PRE-CREATION
bq. flume-ng-sdk/src/main/avro/flume.avdl PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/Event.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/EventDeliveryException.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/NettyAvroRpcClient.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClient.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientFactory.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/event/EventBuilder.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/event/SimpleEvent.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/api/RpcTestUtils.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/api/TestRpcClientFactory.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/event/TestEventBuilder.java PRE-CREATION
bq. flume-ng-sinks/flume-hdfs-sink/pom.xml f27851e
bq. flume-ng-sinks/flume-irc-sink/pom.xml c885f35
bq. pom.xml d785762
bq.
bq. Diff: https://reviews.apache.org/r/4047/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Mike
bq.
bq.
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226602#comment-13226602 ]
jiraposter@reviews.apache.org commented on FLUME-989:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4047/
-----------------------------------------------------------
(Updated 2012-03-09 23:44:00.840039)
Review request for Flume.
Changes
-------
Patch updated. I brought back isAlive() but it basically just guarantees that we haven't seen a major error happen yet.
Summary
-------
Seeking early feedback on some additional APIs to make integrating with Flume 1.x easier.
Added the following APIs:
- AvroClient: Friendly Java interface around the Avro API
- AvroClientBuilder: Builder class to allow easy extension of AvroClient capabilities in the future (i.e. SSL)
- DefaultAvroClient: Implementation of the AvroClient interface
Created this stuff in a flume-ng-sdk Maven submodule and moved the Event interface to that submodule. flume-ng-core depends on flume-ng-sdk.
I also modified AvroSink to use the AvroClient API instead of the bare AvroSourceProtocol API directly.
This addresses bug FLUME-989.
https://issues.apache.org/jira/browse/FLUME-989
Diffs (updated)
-----
flume-ng-clients/flume-ng-log4jappender/pom.xml 9dd31bd
flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java 97f2b9e
flume-ng-core/pom.xml fe6ce0b
flume-ng-core/src/main/avro/flume.avdl 40da3ef
flume-ng-core/src/main/java/org/apache/flume/Event.java 5278fc0
flume-ng-core/src/main/java/org/apache/flume/EventDeliveryException.java 1413223
flume-ng-core/src/main/java/org/apache/flume/FlumeException.java eab5b3d
flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 195ba79
flume-ng-core/src/main/java/org/apache/flume/event/EventBuilder.java 5d8c3b3
flume-ng-core/src/main/java/org/apache/flume/event/SimpleEvent.java e0c3b45
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/util/TestEventBuilder.java 7930607
flume-ng-sdk/pom.xml PRE-CREATION
flume-ng-sdk/src/main/avro/flume.avdl PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/Event.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/EventDeliveryException.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/NettyAvroRpcClient.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClient.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientFactory.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/event/EventBuilder.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/event/SimpleEvent.java PRE-CREATION
flume-ng-sdk/src/test/java/org/apache/flume/api/RpcTestUtils.java PRE-CREATION
flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java PRE-CREATION
flume-ng-sdk/src/test/java/org/apache/flume/api/TestRpcClientFactory.java PRE-CREATION
flume-ng-sdk/src/test/java/org/apache/flume/event/TestEventBuilder.java PRE-CREATION
pom.xml d785762
Diff: https://reviews.apache.org/r/4047/diff
Testing
-------
Thanks,
Mike
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228055#comment-13228055 ]
jiraposter@reviews.apache.org commented on FLUME-989:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4047/
-----------------------------------------------------------
(Updated 2012-03-12 23:33:32.264619)
Review request for Flume.
Changes
-------
Patch updated to allow multiple RPC calls from one batch call if getBatchSize() is not respected. Also added some API docs indicating the behavior and implications of this. I also found a maven configuration param to make the Java generated from Avro RPC use String instead of CharSequence as the String type, which will result in less object copying in our code. I included that, since it primarily affects the RPC client implementation. Finally, one notable API change is that I have declared the NettyAvroRpcClient.Builder class to be protected instead of public. This is to enforce usage of the factory methods in RpcClientFactory, except for in unit tests.
Summary
-------
Seeking early feedback on some additional APIs to make integrating with Flume 1.x easier.
Added the following APIs:
- AvroClient: Friendly Java interface around the Avro API
- AvroClientBuilder: Builder class to allow easy extension of AvroClient capabilities in the future (i.e. SSL)
- DefaultAvroClient: Implementation of the AvroClient interface
Created this stuff in a flume-ng-sdk Maven submodule and moved the Event interface to that submodule. flume-ng-core depends on flume-ng-sdk.
I also modified AvroSink to use the AvroClient API instead of the bare AvroSourceProtocol API directly.
This addresses bug FLUME-989.
https://issues.apache.org/jira/browse/FLUME-989
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/pom.xml ee2d20f
flume-ng-channels/flume-jdbc-channel/pom.xml 51a7694
flume-ng-clients/flume-ng-log4jappender/pom.xml 9dd31bd
flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java 97f2b9e
flume-ng-core/pom.xml fe6ce0b
flume-ng-core/src/main/avro/flume.avdl 40da3ef
flume-ng-core/src/main/java/org/apache/flume/Event.java 5278fc0
flume-ng-core/src/main/java/org/apache/flume/EventDeliveryException.java 1413223
flume-ng-core/src/main/java/org/apache/flume/FlumeException.java eab5b3d
flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 195ba79
flume-ng-core/src/main/java/org/apache/flume/event/EventBuilder.java 5d8c3b3
flume-ng-core/src/main/java/org/apache/flume/event/SimpleEvent.java e0c3b45
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/util/TestEventBuilder.java 7930607
flume-ng-legacy-sources/flume-avro-source/pom.xml 2d53403
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/pom.xml 8256f0f
flume-ng-node/pom.xml b9b062e
flume-ng-sdk/pom.xml PRE-CREATION
flume-ng-sdk/src/main/avro/flume.avdl PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/Event.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/EventDeliveryException.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/NettyAvroRpcClient.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClient.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientFactory.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/event/EventBuilder.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/event/SimpleEvent.java PRE-CREATION
flume-ng-sdk/src/test/java/org/apache/flume/api/RpcTestUtils.java PRE-CREATION
flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java PRE-CREATION
flume-ng-sdk/src/test/java/org/apache/flume/api/TestRpcClientFactory.java PRE-CREATION
flume-ng-sdk/src/test/java/org/apache/flume/event/TestEventBuilder.java PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/pom.xml f27851e
flume-ng-sinks/flume-irc-sink/pom.xml c885f35
pom.xml d785762
Diff: https://reviews.apache.org/r/4047/diff
Testing
-------
Thanks,
Mike
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227720#comment-13227720 ]
jiraposter@reviews.apache.org commented on FLUME-989:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4047/#review5841
-----------------------------------------------------------
Thanks for the patch Mike. I have one suggestion though:
Please modify the default appendBatch() method in the NettyAvroRpcClient() implementation so that it does not throw an IllegalArgumentException if the passed in collection contains more events than the allowed batch size. Instead it should loop over the given collection in chunks of batch sizes until all of it has been drained to the downstream end point.
Reason for this ask is that the builder can potentially use the built in default value of batch size that is not visible to the client directly.
- Arvind
On 2012-03-09 23:44:00, Mike Percy wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4047/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-09 23:44:00)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Seeking early feedback on some additional APIs to make integrating with Flume 1.x easier.
bq.
bq. Added the following APIs:
bq. - AvroClient: Friendly Java interface around the Avro API
bq. - AvroClientBuilder: Builder class to allow easy extension of AvroClient capabilities in the future (i.e. SSL)
bq. - DefaultAvroClient: Implementation of the AvroClient interface
bq.
bq. Created this stuff in a flume-ng-sdk Maven submodule and moved the Event interface to that submodule. flume-ng-core depends on flume-ng-sdk.
bq.
bq. I also modified AvroSink to use the AvroClient API instead of the bare AvroSourceProtocol API directly.
bq.
bq.
bq. This addresses bug FLUME-989.
bq. https://issues.apache.org/jira/browse/FLUME-989
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-clients/flume-ng-log4jappender/pom.xml 9dd31bd
bq. flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java 97f2b9e
bq. flume-ng-core/pom.xml fe6ce0b
bq. flume-ng-core/src/main/avro/flume.avdl 40da3ef
bq. flume-ng-core/src/main/java/org/apache/flume/Event.java 5278fc0
bq. flume-ng-core/src/main/java/org/apache/flume/EventDeliveryException.java 1413223
bq. flume-ng-core/src/main/java/org/apache/flume/FlumeException.java eab5b3d
bq. flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 195ba79
bq. flume-ng-core/src/main/java/org/apache/flume/event/EventBuilder.java 5d8c3b3
bq. flume-ng-core/src/main/java/org/apache/flume/event/SimpleEvent.java e0c3b45
bq. flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
bq. flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
bq. flume-ng-core/src/test/java/org/apache/flume/util/TestEventBuilder.java 7930607
bq. flume-ng-sdk/pom.xml PRE-CREATION
bq. flume-ng-sdk/src/main/avro/flume.avdl PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/Event.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/EventDeliveryException.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/NettyAvroRpcClient.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClient.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientFactory.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/event/EventBuilder.java PRE-CREATION
bq. flume-ng-sdk/src/main/java/org/apache/flume/event/SimpleEvent.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/api/RpcTestUtils.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/api/TestRpcClientFactory.java PRE-CREATION
bq. flume-ng-sdk/src/test/java/org/apache/flume/event/TestEventBuilder.java PRE-CREATION
bq. pom.xml d785762
bq.
bq. Diff: https://reviews.apache.org/r/4047/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Mike
bq.
bq.
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-989) Factor Flume Avro RPC interfaces out
into separate Client SDK
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225140#comment-13225140 ]
jiraposter@reviews.apache.org commented on FLUME-989:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4047/
-----------------------------------------------------------
(Updated 2012-03-08 11:14:53.728421)
Review request for Flume.
Changes
-------
Now applies to merged trunk.
Summary (updated)
-------
Seeking early feedback on some additional APIs to make integrating with Flume 1.x easier.
Added the following APIs:
- AvroClient: Friendly Java interface around the Avro API
- AvroClientBuilder: Builder class to allow easy extension of AvroClient capabilities in the future (i.e. SSL)
- DefaultAvroClient: Implementation of the AvroClient interface
Created this stuff in a flume-ng-sdk Maven submodule and moved the Event interface to that submodule. flume-ng-core depends on flume-ng-sdk.
I also modified AvroSink to use the AvroClient API instead of the bare AvroSourceProtocol API directly.
This addresses bug FLUME-989.
https://issues.apache.org/jira/browse/FLUME-989
Diffs
-----
flume-ng-clients/flume-ng-log4jappender/pom.xml 9dd31bd
flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java 97f2b9e
flume-ng-core/pom.xml fe6ce0b
flume-ng-core/src/main/avro/flume.avdl 40da3ef
flume-ng-core/src/main/java/org/apache/flume/Event.java a017705
flume-ng-core/src/main/java/org/apache/flume/FlumeException.java eab5b3d
flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 195ba79
flume-ng-core/src/main/java/org/apache/flume/event/EventBuilder.java 5d8c3b3
flume-ng-core/src/main/java/org/apache/flume/event/SimpleEvent.java e0c3b45
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-sdk/pom.xml PRE-CREATION
flume-ng-sdk/src/main/avro/flume.avdl PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/Event.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/FlumeRemoteException.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/FlumeTimeoutException.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/client/NettyAvroRpcClient.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcCallback.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClient.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcClientBuilder.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/api/client/RpcResponse.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/event/EventBuilder.java PRE-CREATION
flume-ng-sdk/src/main/java/org/apache/flume/event/SimpleEvent.java PRE-CREATION
pom.xml d785762
Diff: https://reviews.apache.org/r/4047/diff
Testing
-------
Thanks,
Mike
> Factor Flume Avro RPC interfaces out into separate Client SDK
> -------------------------------------------------------------
>
> Key: FLUME-989
> URL: https://issues.apache.org/jira/browse/FLUME-989
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: v1.1.0
>
>
> Factor out the network RPC APIs so that there is a concise API boundary between user and developer APIs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira