You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Yonger (JIRA)" <ji...@apache.org> on 2017/09/21 11:33:00 UTC
[jira] [Resolved] (HADOOP-14770) S3A http connection in s3a driver
not reuse in Spark application
[ https://issues.apache.org/jira/browse/HADOOP-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonger resolved HADOOP-14770.
-----------------------------
Resolution: Duplicate
Thanks Steve. when we apply the read input policy with random in our workload after upgrade to Hadoop 2.8.1, it works as my expect, connections not destroyed every time but reused.
> S3A http connection in s3a driver not reuse in Spark application
> ----------------------------------------------------------------
>
> Key: HADOOP-14770
> URL: https://issues.apache.org/jira/browse/HADOOP-14770
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.7.3
> Reporter: Yonger
> Assignee: Yonger
> Priority: Minor
>
> I print out connection stats every 2 s when running Spark application against s3-compatible storage:
> {code}
> ESTAB 0 0 ::ffff:10.0.2.36:44446 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44454 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44374 ::ffff:10.0.2.254:80
> ESTAB 159724 0 ::ffff:10.0.2.36:44436 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44448 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44338 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44438 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44414 ::ffff:10.0.2.254:80
> ESTAB 0 480 ::ffff:10.0.2.36:44450 ::ffff:10.0.2.254:80 timer:(on,170ms,0)
> ESTAB 0 0 ::ffff:10.0.2.36:44442 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44390 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44326 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44452 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44394 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44444 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44456 ::ffff:10.0.2.254:80
> ======================
> ESTAB 0 0 ::ffff:10.0.2.36:44508 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44476 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44524 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44374 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44500 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44504 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44512 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44506 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44464 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44518 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44510 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44442 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44526 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44472 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44466 ::ffff:10.0.2.254:80
> {code}
> the connection in the above of "=" and below were changed all the time. But this haven't seen in MR application.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org