You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2017/08/14 10:49:00 UTC
[jira] [Updated] (HADOOP-14770) S3A http connection in s3a driver
not reuse in Spark application
[ https://issues.apache.org/jira/browse/HADOOP-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-14770:
------------------------------------
Description:
I print out connection stats every 2 s when running Spark application against s3-compatible storage:
{code}
ESTAB 0 0 ::ffff:10.0.2.36:44446 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44454 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44374 ::ffff:10.0.2.254:80
ESTAB 159724 0 ::ffff:10.0.2.36:44436 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44448 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44338 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44438 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44414 ::ffff:10.0.2.254:80
ESTAB 0 480 ::ffff:10.0.2.36:44450 ::ffff:10.0.2.254:80 timer:(on,170ms,0)
ESTAB 0 0 ::ffff:10.0.2.36:44442 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44390 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44326 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44452 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44394 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44444 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44456 ::ffff:10.0.2.254:80
======================
ESTAB 0 0 ::ffff:10.0.2.36:44508 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44476 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44524 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44374 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44500 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44504 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44512 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44506 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44464 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44518 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44510 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44442 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44526 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44472 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44466 ::ffff:10.0.2.254:80
{code}
the connection in the above of "=" and below were changed all the time. But this haven't seen in MR application.
was:
I print out connection stats every 2 s when running Spark application against s3-compatible storage:
ESTAB 0 0 ::ffff:10.0.2.36:44446 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44454 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44374 ::ffff:10.0.2.254:80
ESTAB 159724 0 ::ffff:10.0.2.36:44436 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44448 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44338 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44438 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44414 ::ffff:10.0.2.254:80
ESTAB 0 480 ::ffff:10.0.2.36:44450 ::ffff:10.0.2.254:80 timer:(on,170ms,0)
ESTAB 0 0 ::ffff:10.0.2.36:44442 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44390 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44326 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44452 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44394 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44444 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44456 ::ffff:10.0.2.254:80
======================
ESTAB 0 0 ::ffff:10.0.2.36:44508 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44476 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44524 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44374 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44500 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44504 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44512 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44506 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44464 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44518 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44510 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44442 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44526 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44472 ::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44466 ::ffff:10.0.2.254:80
the connection in the above of "=" and below were changed all the time. But this haven't seen in MR application.
> S3A http connection in s3a driver not reuse in Spark application
> ----------------------------------------------------------------
>
> Key: HADOOP-14770
> URL: https://issues.apache.org/jira/browse/HADOOP-14770
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Yonger
> Assignee: Yonger
>
> I print out connection stats every 2 s when running Spark application against s3-compatible storage:
> {code}
> ESTAB 0 0 ::ffff:10.0.2.36:44446 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44454 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44374 ::ffff:10.0.2.254:80
> ESTAB 159724 0 ::ffff:10.0.2.36:44436 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44448 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44338 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44438 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44414 ::ffff:10.0.2.254:80
> ESTAB 0 480 ::ffff:10.0.2.36:44450 ::ffff:10.0.2.254:80 timer:(on,170ms,0)
> ESTAB 0 0 ::ffff:10.0.2.36:44442 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44390 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44326 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44452 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44394 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44444 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44456 ::ffff:10.0.2.254:80
> ======================
> ESTAB 0 0 ::ffff:10.0.2.36:44508 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44476 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44524 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44374 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44500 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44504 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44512 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44506 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44464 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44518 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44510 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44442 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44526 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44472 ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44466 ::ffff:10.0.2.254:80
> {code}
> the connection in the above of "=" and below were changed all the time. But this haven't seen in MR application.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org