You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Arvind Prabhakar <ar...@apache.org> on 2012/04/12 02:30:21 UTC
Review Request: FLUME-1113. JDBC channel invokes size query on every put
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4704/
-----------------------------------------------------------
Review request for Flume.
Summary
-------
This change modifies the JDBC channel provider implementation so that it invokes the size query only during configuration to initialize it's own execution state. At runtime, the transaction instances keep track of how many events get added and removed and the delta is then added to the size maintained by the provider. Some notes about this implementation:
* It is possible that between the commit and update of the size counter by one transaction, another transaction may read stale size information.
* The size counter is only used to gate put operations and not take. This ensures that stale or incorrect information does not inhibit the draining of the channel.
Also refactored the channel tests to be run with and without foreign keys enabled.
This addresses bug FLUME-1113.
https://issues.apache.org/jira/browse/FLUME-1113
Diffs
-----
/trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1325037
/trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionFactory.java 1325037
/trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java 1325037
/trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/BaseJdbcChannelProviderTest.java PRE-CREATION
/trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 1325037
/trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProviderNoFK.java PRE-CREATION
Diff: https://reviews.apache.org/r/4704/diff
Testing
-------
Full build and tests.
Thanks,
Arvind
Re: Review Request: FLUME-1113. JDBC channel invokes size query on every put
Posted by Arvind Prabhakar <ar...@apache.org>.
> On 2012-04-12 01:37:23, Prasad Mujumdar wrote:
> > /trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java, line 304
> > <https://reviews.apache.org/r/4704/diff/2/?file=101399#file101399line304>
> >
> > should be currentSizeLong
Done.
> On 2012-04-12 01:37:23, Prasad Mujumdar wrote:
> > /trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java, line 127
> > <https://reviews.apache.org/r/4704/diff/2/?file=101401#file101401line127>
> >
> > Do we need to reset these new counters in close() ?
It does not matter as the transaction instance is discarded after this point. However, I have incorporated your suggestion.
- Arvind
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4704/#review6865
-----------------------------------------------------------
On 2012-04-12 01:13:25, Arvind Prabhakar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4704/
> -----------------------------------------------------------
>
> (Updated 2012-04-12 01:13:25)
>
>
> Review request for Flume.
>
>
> Summary
> -------
>
> This change modifies the JDBC channel provider implementation so that it invokes the size query only during configuration to initialize it's own execution state. At runtime, the transaction instances keep track of how many events get added and removed and the delta is then added to the size maintained by the provider. Some notes about this implementation:
>
> * It is possible that between the commit and update of the size counter by one transaction, another transaction may read stale size information.
> * The size counter is only used to gate put operations and not take. This ensures that stale or incorrect information does not inhibit the draining of the channel.
>
> Also refactored the channel tests to be run with and without foreign keys enabled.
>
>
> This addresses bug FLUME-1113.
> https://issues.apache.org/jira/browse/FLUME-1113
>
>
> Diffs
> -----
>
> /trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1325037
> /trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionFactory.java 1325037
> /trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java 1325037
> /trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/BaseJdbcChannelProviderTest.java PRE-CREATION
> /trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 1325037
> /trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProviderNoFK.java PRE-CREATION
> /trunk/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1325037
>
> Diff: https://reviews.apache.org/r/4704/diff
>
>
> Testing
> -------
>
> Full build and tests.
>
>
> Thanks,
>
> Arvind
>
>
Re: Review Request: FLUME-1113. JDBC channel invokes size query on every put
Posted by Prasad Mujumdar <pr...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4704/#review6865
-----------------------------------------------------------
Ship it!
lgtm.
/trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java
<https://reviews.apache.org/r/4704/#comment15289>
should be currentSizeLong
/trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java
<https://reviews.apache.org/r/4704/#comment15290>
Do we need to reset these new counters in close() ?
- Prasad
On 2012-04-12 01:13:25, Arvind Prabhakar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4704/
> -----------------------------------------------------------
>
> (Updated 2012-04-12 01:13:25)
>
>
> Review request for Flume.
>
>
> Summary
> -------
>
> This change modifies the JDBC channel provider implementation so that it invokes the size query only during configuration to initialize it's own execution state. At runtime, the transaction instances keep track of how many events get added and removed and the delta is then added to the size maintained by the provider. Some notes about this implementation:
>
> * It is possible that between the commit and update of the size counter by one transaction, another transaction may read stale size information.
> * The size counter is only used to gate put operations and not take. This ensures that stale or incorrect information does not inhibit the draining of the channel.
>
> Also refactored the channel tests to be run with and without foreign keys enabled.
>
>
> This addresses bug FLUME-1113.
> https://issues.apache.org/jira/browse/FLUME-1113
>
>
> Diffs
> -----
>
> /trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1325037
> /trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionFactory.java 1325037
> /trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java 1325037
> /trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/BaseJdbcChannelProviderTest.java PRE-CREATION
> /trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 1325037
> /trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProviderNoFK.java PRE-CREATION
> /trunk/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1325037
>
> Diff: https://reviews.apache.org/r/4704/diff
>
>
> Testing
> -------
>
> Full build and tests.
>
>
> Thanks,
>
> Arvind
>
>
Re: Review Request: FLUME-1113. JDBC channel invokes size query on every put
Posted by Arvind Prabhakar <ar...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4704/
-----------------------------------------------------------
(Updated 2012-04-12 03:11:49.860724)
Review request for Flume.
Changes
-------
Incorporated review feedback.
Summary
-------
This change modifies the JDBC channel provider implementation so that it invokes the size query only during configuration to initialize it's own execution state. At runtime, the transaction instances keep track of how many events get added and removed and the delta is then added to the size maintained by the provider. Some notes about this implementation:
* It is possible that between the commit and update of the size counter by one transaction, another transaction may read stale size information.
* The size counter is only used to gate put operations and not take. This ensures that stale or incorrect information does not inhibit the draining of the channel.
Also refactored the channel tests to be run with and without foreign keys enabled.
This addresses bug FLUME-1113.
https://issues.apache.org/jira/browse/FLUME-1113
Diffs (updated)
-----
/trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1325037
/trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionFactory.java 1325037
/trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java 1325037
/trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/BaseJdbcChannelProviderTest.java PRE-CREATION
/trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 1325037
/trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProviderNoFK.java PRE-CREATION
/trunk/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1325037
Diff: https://reviews.apache.org/r/4704/diff
Testing
-------
Full build and tests.
Thanks,
Arvind
Re: Review Request: FLUME-1113. JDBC channel invokes size query on every put
Posted by Arvind Prabhakar <ar...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4704/
-----------------------------------------------------------
(Updated 2012-04-12 01:13:25.731663)
Review request for Flume.
Changes
-------
During manual testing, I realized that the avro source was not logging channel exception. Added a log message to facilitate testing and also removed some empty lines. No logic changes in this diff update.
Summary
-------
This change modifies the JDBC channel provider implementation so that it invokes the size query only during configuration to initialize it's own execution state. At runtime, the transaction instances keep track of how many events get added and removed and the delta is then added to the size maintained by the provider. Some notes about this implementation:
* It is possible that between the commit and update of the size counter by one transaction, another transaction may read stale size information.
* The size counter is only used to gate put operations and not take. This ensures that stale or incorrect information does not inhibit the draining of the channel.
Also refactored the channel tests to be run with and without foreign keys enabled.
This addresses bug FLUME-1113.
https://issues.apache.org/jira/browse/FLUME-1113
Diffs (updated)
-----
/trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1325037
/trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionFactory.java 1325037
/trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java 1325037
/trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/BaseJdbcChannelProviderTest.java PRE-CREATION
/trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 1325037
/trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProviderNoFK.java PRE-CREATION
/trunk/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1325037
Diff: https://reviews.apache.org/r/4704/diff
Testing
-------
Full build and tests.
Thanks,
Arvind
Filtering messages
Posted by Ralph Goers <ra...@dslextreme.com>.
I'm trying to get rid of test messages from my flume-dev folder so that I can reduce the noise to only important stuff - i.e messages posted by humans. Does anyone know how to filter these in gmail?
Ralph
On Apr 11, 2012, at 5:30 PM, Arvind Prabhakar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4704/
> -----------------------------------------------------------
>
> Review request for Flume.
>
>
> Summary
> -------
>
> This change modifies the JDBC channel provider implementation so that it invokes the size query only during configuration to initialize it's own execution state. At runtime, the transaction instances keep track of how many events get added and removed and the delta is then added to the size maintained by the provider. Some notes about this implementation:
>
> * It is possible that between the commit and update of the size counter by one transaction, another transaction may read stale size information.
> * The size counter is only used to gate put operations and not take. This ensures that stale or incorrect information does not inhibit the draining of the channel.
>
> Also refactored the channel tests to be run with and without foreign keys enabled.
>
>
> This addresses bug FLUME-1113.
> https://issues.apache.org/jira/browse/FLUME-1113
>
>
> Diffs
> -----
>
> /trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1325037
> /trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionFactory.java 1325037
> /trunk/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java 1325037
> /trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/BaseJdbcChannelProviderTest.java PRE-CREATION
> /trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 1325037
> /trunk/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProviderNoFK.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/4704/diff
>
>
> Testing
> -------
>
> Full build and tests.
>
>
> Thanks,
>
> Arvind
>