You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Mike Percy (Created) (JIRA)" <ji...@apache.org> on 2012/03/01 03:27:58 UTC
[jira] [Created] (FLUME-1011) AvroSource should have a configurable
max thread count
AvroSource should have a configurable max thread count
------------------------------------------------------
Key: FLUME-1011
URL: https://issues.apache.org/jira/browse/FLUME-1011
Project: Flume
Issue Type: Improvement
Affects Versions: v1.0.0
Reporter: Mike Percy
Fix For: v1.1.0
AvroSource via NettyServer uses Executors.newCachedThreadPool() when accepting incoming connections, which means that there is no upper limit on the number of threads that can be spawned. It would be good to expose configurable core & max thread pool sizes and provide some sane defaults to avoid OutOfMemory errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (FLUME-1011) AvroSource should have a
configurable max thread count
Posted by "Brock Noland (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland resolved FLUME-1011.
---------------------------------
Resolution: Fixed
> AvroSource should have a configurable max thread count
> ------------------------------------------------------
>
> Key: FLUME-1011
> URL: https://issues.apache.org/jira/browse/FLUME-1011
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Juhani Connolly
> Fix For: v1.2.0
>
> Attachments: FLUME-1011.2.patch
>
>
> AvroSource via NettyServer uses Executors.newCachedThreadPool() when accepting incoming connections, which means that there is no upper limit on the number of threads that can be spawned. It would be good to expose configurable core & max thread pool sizes and provide some sane defaults to avoid OutOfMemory errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1011) AvroSource should have a
configurable max thread count
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249768#comment-13249768 ]
jiraposter@reviews.apache.org commented on FLUME-1011:
------------------------------------------------------
bq. On 2012-04-06 14:18:15, Brock Noland wrote:
bq. > flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java, line 123
bq. > <https://reviews.apache.org/r/4669/diff/2/?file=100629#file100629line123>
bq. >
bq. > context.getInteger should be used here
I forgot we had those now! done
- Juhani
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4669/#review6742
-----------------------------------------------------------
On 2012-04-09 10:07:06, Juhani Connolly wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4669/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-04-09 10:07:06)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. A rudimentary fix allowing setting a thread limit to the netty server used for avro ipc.
bq.
bq. A more involved solution would be to implement a custom thread pool but I don't think that is entirely necessary
bq.
bq.
bq. This addresses bug FLUME-1011.
bq. https://issues.apache.org/jira/browse/FLUME-1011
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
bq. flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
bq.
bq. Diff: https://reviews.apache.org/r/4669/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Added the new setting into one of the tests to make sure it is exercised
bq.
bq. Tests pass individually, I'm still having issues with running several of the RPC related tests from mvn, but this was the case before adding this code.
bq.
bq.
bq. Thanks,
bq.
bq. Juhani
bq.
bq.
> AvroSource should have a configurable max thread count
> ------------------------------------------------------
>
> Key: FLUME-1011
> URL: https://issues.apache.org/jira/browse/FLUME-1011
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Fix For: v1.2.0
>
>
> AvroSource via NettyServer uses Executors.newCachedThreadPool() when accepting incoming connections, which means that there is no upper limit on the number of threads that can be spawned. It would be good to expose configurable core & max thread pool sizes and provide some sane defaults to avoid OutOfMemory errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1011) AvroSource should have a
configurable max thread count
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249771#comment-13249771 ]
jiraposter@reviews.apache.org commented on FLUME-1011:
------------------------------------------------------
bq. On 2012-04-07 12:57:18, Will McQueen wrote:
bq. > flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java, line 125
bq. > <https://reviews.apache.org/r/4669/diff/2/?file=100629#file100629line125>
bq. >
bq. > Recommend this warning string:
bq. > AVRO source\'s \"threads\" prop must specify integer value.
bq. >
bq. > (here we're using the name of the type instead of the class name)
modified
- Juhani
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4669/#review6771
-----------------------------------------------------------
On 2012-04-09 10:07:06, Juhani Connolly wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4669/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-04-09 10:07:06)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. A rudimentary fix allowing setting a thread limit to the netty server used for avro ipc.
bq.
bq. A more involved solution would be to implement a custom thread pool but I don't think that is entirely necessary
bq.
bq.
bq. This addresses bug FLUME-1011.
bq. https://issues.apache.org/jira/browse/FLUME-1011
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
bq. flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
bq.
bq. Diff: https://reviews.apache.org/r/4669/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Added the new setting into one of the tests to make sure it is exercised
bq.
bq. Tests pass individually, I'm still having issues with running several of the RPC related tests from mvn, but this was the case before adding this code.
bq.
bq.
bq. Thanks,
bq.
bq. Juhani
bq.
bq.
> AvroSource should have a configurable max thread count
> ------------------------------------------------------
>
> Key: FLUME-1011
> URL: https://issues.apache.org/jira/browse/FLUME-1011
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Fix For: v1.2.0
>
> Attachments: FLUME-1011.2.patch
>
>
> AvroSource via NettyServer uses Executors.newCachedThreadPool() when accepting incoming connections, which means that there is no upper limit on the number of threads that can be spawned. It would be good to expose configurable core & max thread pool sizes and provide some sane defaults to avoid OutOfMemory errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1011) AvroSource should have a
configurable max thread count
Posted by "Juhani Connolly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248140#comment-13248140 ]
Juhani Connolly commented on FLUME-1011:
----------------------------------------
Since no-one seemed to be working on this, I put together a basic patch that addresses the core issue of potentially spawning too many threads.
If the core thread pool size needs to be configured too, maybe that could be turned into a separate issue?
> AvroSource should have a configurable max thread count
> ------------------------------------------------------
>
> Key: FLUME-1011
> URL: https://issues.apache.org/jira/browse/FLUME-1011
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Fix For: v1.2.0
>
>
> AvroSource via NettyServer uses Executors.newCachedThreadPool() when accepting incoming connections, which means that there is no upper limit on the number of threads that can be spawned. It would be good to expose configurable core & max thread pool sizes and provide some sane defaults to avoid OutOfMemory errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1011) AvroSource should have a
configurable max thread count
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248388#comment-13248388 ]
jiraposter@reviews.apache.org commented on FLUME-1011:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4669/#review6742
-----------------------------------------------------------
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java
<https://reviews.apache.org/r/4669/#comment14728>
context.getInteger should be used here
- Brock
On 2012-04-06 06:51:59, Juhani Connolly wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4669/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-04-06 06:51:59)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. A rudimentary fix allowing setting a thread limit to the netty server used for avro ipc.
bq.
bq. A more involved solution would be to implement a custom thread pool but I don't think that is entirely necessary
bq.
bq.
bq. This addresses bug FLUME-1011.
bq. https://issues.apache.org/jira/browse/FLUME-1011
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
bq. flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
bq.
bq. Diff: https://reviews.apache.org/r/4669/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Added the new setting into one of the tests to make sure it is exercised
bq.
bq. Tests pass individually, I'm still having issues with running several of the RPC related tests from mvn, but this was the case before adding this code.
bq.
bq.
bq. Thanks,
bq.
bq. Juhani
bq.
bq.
> AvroSource should have a configurable max thread count
> ------------------------------------------------------
>
> Key: FLUME-1011
> URL: https://issues.apache.org/jira/browse/FLUME-1011
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Fix For: v1.2.0
>
>
> AvroSource via NettyServer uses Executors.newCachedThreadPool() when accepting incoming connections, which means that there is no upper limit on the number of threads that can be spawned. It would be good to expose configurable core & max thread pool sizes and provide some sane defaults to avoid OutOfMemory errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1011) AvroSource should have a configurable
max thread count
Posted by "Juhani Connolly (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Juhani Connolly updated FLUME-1011:
-----------------------------------
Attachment: FLUME-1011.2.patch
Adding the patch from the related review
> AvroSource should have a configurable max thread count
> ------------------------------------------------------
>
> Key: FLUME-1011
> URL: https://issues.apache.org/jira/browse/FLUME-1011
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Fix For: v1.2.0
>
> Attachments: FLUME-1011.2.patch
>
>
> AvroSource via NettyServer uses Executors.newCachedThreadPool() when accepting incoming connections, which means that there is no upper limit on the number of threads that can be spawned. It would be good to expose configurable core & max thread pool sizes and provide some sane defaults to avoid OutOfMemory errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1011) AvroSource should have a
configurable max thread count
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250288#comment-13250288 ]
jiraposter@reviews.apache.org commented on FLUME-1011:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4669/#review6809
-----------------------------------------------------------
Ship it!
lgtm
- Brock
On 2012-04-09 10:07:06, Juhani Connolly wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4669/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-04-09 10:07:06)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. A rudimentary fix allowing setting a thread limit to the netty server used for avro ipc.
bq.
bq. A more involved solution would be to implement a custom thread pool but I don't think that is entirely necessary
bq.
bq.
bq. This addresses bug FLUME-1011.
bq. https://issues.apache.org/jira/browse/FLUME-1011
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
bq. flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
bq.
bq. Diff: https://reviews.apache.org/r/4669/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Added the new setting into one of the tests to make sure it is exercised
bq.
bq. Tests pass individually, I'm still having issues with running several of the RPC related tests from mvn, but this was the case before adding this code.
bq.
bq.
bq. Thanks,
bq.
bq. Juhani
bq.
bq.
> AvroSource should have a configurable max thread count
> ------------------------------------------------------
>
> Key: FLUME-1011
> URL: https://issues.apache.org/jira/browse/FLUME-1011
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Fix For: v1.2.0
>
> Attachments: FLUME-1011.2.patch
>
>
> AvroSource via NettyServer uses Executors.newCachedThreadPool() when accepting incoming connections, which means that there is no upper limit on the number of threads that can be spawned. It would be good to expose configurable core & max thread pool sizes and provide some sane defaults to avoid OutOfMemory errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1011) AvroSource should have a
configurable max thread count
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248137#comment-13248137 ]
jiraposter@reviews.apache.org commented on FLUME-1011:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4669/
-----------------------------------------------------------
Review request for Flume.
Summary
-------
A rudimentary fix allowing setting a thread limit to the netty server used for avro ipc.
A more involved solution would be to implement a custom thread pool but I don't think that is entirely necessary
This addresses bug FLUME-1011.
https://issues.apache.org/jira/browse/FLUME-1011
Diffs
-----
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
Diff: https://reviews.apache.org/r/4669/diff
Testing
-------
Added the new setting into one of the tests to make sure it is exercised
Tests pass individually, I'm still having issues with running several of the RPC related tests from mvn, but this was the case before adding this code.
Thanks,
Juhani
> AvroSource should have a configurable max thread count
> ------------------------------------------------------
>
> Key: FLUME-1011
> URL: https://issues.apache.org/jira/browse/FLUME-1011
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Fix For: v1.2.0
>
>
> AvroSource via NettyServer uses Executors.newCachedThreadPool() when accepting incoming connections, which means that there is no upper limit on the number of threads that can be spawned. It would be good to expose configurable core & max thread pool sizes and provide some sane defaults to avoid OutOfMemory errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1011) AvroSource should have a
configurable max thread count
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249241#comment-13249241 ]
jiraposter@reviews.apache.org commented on FLUME-1011:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4669/#review6771
-----------------------------------------------------------
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java
<https://reviews.apache.org/r/4669/#comment14929>
Recommend this warning string:
AVRO source\'s \"threads\" prop must specify integer value.
(here we're using the name of the type instead of the class name)
- Will
On 2012-04-06 06:51:59, Juhani Connolly wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4669/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-04-06 06:51:59)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. A rudimentary fix allowing setting a thread limit to the netty server used for avro ipc.
bq.
bq. A more involved solution would be to implement a custom thread pool but I don't think that is entirely necessary
bq.
bq.
bq. This addresses bug FLUME-1011.
bq. https://issues.apache.org/jira/browse/FLUME-1011
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
bq. flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
bq.
bq. Diff: https://reviews.apache.org/r/4669/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Added the new setting into one of the tests to make sure it is exercised
bq.
bq. Tests pass individually, I'm still having issues with running several of the RPC related tests from mvn, but this was the case before adding this code.
bq.
bq.
bq. Thanks,
bq.
bq. Juhani
bq.
bq.
> AvroSource should have a configurable max thread count
> ------------------------------------------------------
>
> Key: FLUME-1011
> URL: https://issues.apache.org/jira/browse/FLUME-1011
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Fix For: v1.2.0
>
>
> AvroSource via NettyServer uses Executors.newCachedThreadPool() when accepting incoming connections, which means that there is no upper limit on the number of threads that can be spawned. It would be good to expose configurable core & max thread pool sizes and provide some sane defaults to avoid OutOfMemory errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (FLUME-1011) AvroSource should have a
configurable max thread count
Posted by "Brock Noland (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland reassigned FLUME-1011:
-----------------------------------
Assignee: Juhani Connolly
Committed in 1311535. Thank you for your contribution!
> AvroSource should have a configurable max thread count
> ------------------------------------------------------
>
> Key: FLUME-1011
> URL: https://issues.apache.org/jira/browse/FLUME-1011
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Juhani Connolly
> Fix For: v1.2.0
>
> Attachments: FLUME-1011.2.patch
>
>
> AvroSource via NettyServer uses Executors.newCachedThreadPool() when accepting incoming connections, which means that there is no upper limit on the number of threads that can be spawned. It would be good to expose configurable core & max thread pool sizes and provide some sane defaults to avoid OutOfMemory errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1011) AvroSource should have a
configurable max thread count
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250365#comment-13250365 ]
Hudson commented on FLUME-1011:
-------------------------------
Integrated in flume-trunk #167 (See [https://builds.apache.org/job/flume-trunk/167/])
FLUME-1011: AvroSource should have a configurable max thread count
(Juhani Connolly via Brock Noland) (Revision 1311535)
Result = SUCCESS
brock : http://svn.apache.org/viewvc/?view=rev&rev=1311535
Files :
* /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java
* /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java
> AvroSource should have a configurable max thread count
> ------------------------------------------------------
>
> Key: FLUME-1011
> URL: https://issues.apache.org/jira/browse/FLUME-1011
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Assignee: Juhani Connolly
> Fix For: v1.2.0
>
> Attachments: FLUME-1011.2.patch
>
>
> AvroSource via NettyServer uses Executors.newCachedThreadPool() when accepting incoming connections, which means that there is no upper limit on the number of threads that can be spawned. It would be good to expose configurable core & max thread pool sizes and provide some sane defaults to avoid OutOfMemory errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1011) AvroSource should have a
configurable max thread count
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249769#comment-13249769 ]
jiraposter@reviews.apache.org commented on FLUME-1011:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4669/
-----------------------------------------------------------
(Updated 2012-04-09 10:07:06.560520)
Review request for Flume.
Changes
-------
Addressed suggestions, added javadoc for property
Summary
-------
A rudimentary fix allowing setting a thread limit to the netty server used for avro ipc.
A more involved solution would be to implement a custom thread pool but I don't think that is entirely necessary
This addresses bug FLUME-1011.
https://issues.apache.org/jira/browse/FLUME-1011
Diffs (updated)
-----
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
Diff: https://reviews.apache.org/r/4669/diff
Testing
-------
Added the new setting into one of the tests to make sure it is exercised
Tests pass individually, I'm still having issues with running several of the RPC related tests from mvn, but this was the case before adding this code.
Thanks,
Juhani
> AvroSource should have a configurable max thread count
> ------------------------------------------------------
>
> Key: FLUME-1011
> URL: https://issues.apache.org/jira/browse/FLUME-1011
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Mike Percy
> Fix For: v1.2.0
>
>
> AvroSource via NettyServer uses Executors.newCachedThreadPool() when accepting incoming connections, which means that there is no upper limit on the number of threads that can be spawned. It would be good to expose configurable core & max thread pool sizes and provide some sane defaults to avoid OutOfMemory errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira