You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Thomas Demoor (JIRA)" <ji...@apache.org> on 2015/08/11 17:57:46 UTC
[jira] [Updated] (HADOOP-11684) S3a to use thread pool that blocks
clients
[ https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Demoor updated HADOOP-11684:
-----------------------------------
Attachment: HADOOP-11684-002.patch
The aws-sdk version bump was seperated out to HADOOP-12269 and merged into trunk. This patch only contains the BlockingThreadpool code.
To witness the exception without the fix:
* only apply the new testclass: {{TestS3ABlockingThreadpool.java}}
* manually add a line setting CORE_THREADS to 2: (needed because this constant is removed in the patch)
{code}
@Before
public void setUp() throws Exception {
conf = new Configuration();
...
conf.setInt(Constants.MAX_TOTAL_TASKS, 1);
+ conf.setInt(Constants.CORE_THREADS, 2);
}
{code}
* run test: The upload will generate 4 parts which exceeds threads + tasks -> Exception
> S3a to use thread pool that blocks clients
> ------------------------------------------
>
> Key: HADOOP-11684
> URL: https://issues.apache.org/jira/browse/HADOOP-11684
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.7.0
> Reporter: Thomas Demoor
> Assignee: Thomas Demoor
> Attachments: HADOOP-11684-001.patch, HADOOP-11684-002.patch
>
>
> Currently, if fs.s3a.max.total.tasks are queued and another (part)upload wants to start, a RejectedExecutionException is thrown.
> We should use a threadpool that blocks clients, nicely throtthling them, rather than throwing an exception. F.i. something similar to https://github.com/apache/incubator-s4/blob/master/subprojects/s4-comm/src/main/java/org/apache/s4/comm/staging/BlockingThreadPoolExecutorService.java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)