[jira] [Commented] (HADOOP-15729) [s3a] stop treat fs.s3a.max.threads as the long-term minimum

["Sean Mackrory (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-15729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871830#comment-16871830 ] 

Sean Mackrory commented on HADOOP-15729:
----------------------------------------

Attach a patch with my proposed change. Unfortunately, it has a bigger impact on simple things like 1 GB upload than I thought, although it's hard to be sure it's not noise. See below for numbers to upload a 1GB file from a machine in us-west-2 to a bucket / pre-created DynamoDB table in the same region. Maybe this is worth adding fs.s3a.core.threads with a moderate default value. Long-running processes (like Hive Server2) might access many buckets and fs.s3a.max.threads grows absolutely out of control - core threads could still do the same unless it's *much* lower, in which case you'd easily hit this performance regression anyway. I would suggest we just proceed and consider fs.s3a.core.threads if further performance testing reveals an issue. Thoughts?

Without change:
{code}
real	0m27.415s
user	0m25.128s
sys	0m6.377s

real	0m25.360s
user	0m25.081s
sys	0m6.368s

real	0m27.615s
user	0m25.296s
sys	0m6.015s

real	0m25.001s
user	0m25.408s
sys	0m6.717s

real	0m28.083s
user	0m24.764s
sys	0m5.774s

real	0m26.117s
user	0m25.192s
sys	0m5.867s
{code}

With change:
{code}
real	0m28.928s
user	0m24.182s
sys	0m5.699s

real	0m33.359s
user	0m25.508s
sys	0m6.407s

real	0m44.412s
user	0m24.565s
sys	0m6.226s

real	0m27.469s
user	0m25.326s
sys	0m6.142s

real	0m35.660s
user	0m25.206s
sys	0m6.154s

real	0m31.811s
user	0m25.042s
sys	0m6.057s
{code}

> [s3a] stop treat fs.s3a.max.threads as the long-term minimum
> ------------------------------------------------------------
>
>                 Key: HADOOP-15729
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15729
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Sean Mackrory
>            Assignee: Sean Mackrory
>            Priority: Major
>         Attachments: HADOOP-15729.001.patch
>
>
> A while ago the s3a connector started experiencing deadlocks because the AWS SDK requires an unbounded threadpool. It places monitoring tasks on the work queue before the tasks they wait on, so it's possible (has even happened with larger-than-default threadpools) for the executor to become permanently saturated and deadlock.
> So we started giving an unbounded threadpool executor to the SDK, and using a bounded, blocking threadpool service for everything else S3A needs (although currently that's only in the S3ABlockOutputStream). fs.s3a.max.threads then only limits this threadpool, however we also specified fs.s3a.max.threads as the number of core threads in the unbounded threadpool, which in hindsight is pretty terrible.
> Currently those core threads do not timeout, so this is actually setting a sort of minimum. Once that many tasks have been submitted, the threadpool will be locked at that number until it bursts beyond that, but it will only spin down that far. If fs.s3a.max.threads is set reasonably high and someone uses a bunch of S3 buckets, they could easily have thousands of idle threads constantly.
> We should either not use fs.s3a.max.threads for the corepool size and introduce a new configuration, or we should simply allow core threads to timeout. I'm reading the OpenJDK source now to see what subtle differences there are between core threads and other threads if core threads can timeout.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org