You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sa Xiao <sa...@gmail.com> on 2015/12/07 09:10:21 UTC
Task hung on SocketInputStream.socketRead0 when reading large a mount
of data from AWS S3
Hi,
We encounter a problem very similar to this one:
https://www.mail-archive.com/search?l=user@spark.apache.org&q=subject:%22Spark+task+hangs+infinitely+when+accessing+S3+from+AWS%22&o=newest&f=1
When reading large amount of data from S3, one or several tasks hung. It
doesn't happen every time, but pretty consistently about at least 1 out of
3 times.
Spark 1.5
mesos slaves: 40 amazon 3r.xlarge (4 core, 30 GB) machines.
total data read from S3: ~380 GB
*spark config that's not default:*
spark.mesos.coarse = true
--conf spark.sql.shuffle.partitions=300
--conf spark.executor.memory=25G
--conf spark.sql.tungsten.enabled=false
*The thread dump of the hanging task:*
Executor task launch worker-3[1] where
[1] java.net.SocketInputStream.socketRead0 (native method)
[2] java.net.SocketInputStream.socketRead (SocketInputStream.java:116)
[3] java.net.SocketInputStream.read (SocketInputStream.java:170)
[4] java.net.SocketInputStream.read (SocketInputStream.java:141)
[5] sun.security.ssl.InputRecord.readFully (InputRecord.java:465)
[6] sun.security.ssl.InputRecord.read (InputRecord.java:503)
[7] sun.security.ssl.SSLSocketImpl.readRecord (SSLSocketImpl.java:961)
[8] sun.security.ssl.SSLSocketImpl.performInitialHandshake
(SSLSocketImpl.java:1,363)
[9] sun.security.ssl.SSLSocketImpl.startHandshake
(SSLSocketImpl.java:1,391)
[10] sun.security.ssl.SSLSocketImpl.startHandshake
(SSLSocketImpl.java:1,375)
[11] org.apache.http.conn.ssl.SSLSocketFactory.connectSocket
(SSLSocketFactory.java:533)
[12] org.apache.http.conn.ssl.SSLSocketFactory.connectSocket
(SSLSocketFactory.java:401)
[13]
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection
(DefaultClientConnectionOperator.java:177)
[14] org.apache.http.impl.conn.ManagedClientConnectionImpl.open
(ManagedClientConnectionImpl.java:304)
[15] org.apache.http.impl.client.DefaultRequestDirector.tryConnect
(DefaultRequestDirector.java:610)
[16] org.apache.http.impl.client.DefaultRequestDirector.execute
(DefaultRequestDirector.java:445)
[17] org.apache.http.impl.client.AbstractHttpClient.doExecute
(AbstractHttpClient.java:863)
[18] org.apache.http.impl.client.CloseableHttpClient.execute
(CloseableHttpClient.java:82)
[19] org.apache.http.impl.client.CloseableHttpClient.execute
(CloseableHttpClient.java:57)
[20] com.amazonaws.http.AmazonHttpClient.executeHelper
(AmazonHttpClient.java:384)
[21] com.amazonaws.http.AmazonHttpClient.execute
(AmazonHttpClient.java:232)
[22] com.amazonaws.services.s3.AmazonS3Client.invoke
(AmazonS3Client.java:3,528)
[23] com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata
(AmazonS3Client.java:976)
[24] com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata
(AmazonS3Client.java:956)
[25] org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus
(S3AFileSystem.java:892)
[26] org.apache.hadoop.fs.s3a.S3AFileSystem.open (S3AFileSystem.java:373)
[27] org.apache.hadoop.fs.FileSystem.open (FileSystem.java:711)
[28] org.apache.hadoop.mapred.LineRecordReader.<init>
(LineRecordReader.java:93)
[29] org.apache.hadoop.mapred.TextInputFormat.getRecordReader
(TextInputFormat.java:54)
[30] org.apache.spark.rdd.HadoopRDD$$anon$1.<init> (HadoopRDD.scala:239)
[31] org.apache.spark.rdd.HadoopRDD.compute (HadoopRDD.scala:216)
[32] org.apache.spark.rdd.HadoopRDD.compute (HadoopRDD.scala:101)
[33] org.apache.spark.rdd.RDD.computeOrReadCheckpoint (RDD.scala:297)
[34] org.apache.spark.rdd.RDD.iterator (RDD.scala:264)
[35] org.apache.spark.rdd.MapPartitionsRDD.compute
(MapPartitionsRDD.scala:38)
[36] org.apache.spark.rdd.RDD.computeOrReadCheckpoint (RDD.scala:297)
[37] org.apache.spark.rdd.RDD.iterator (RDD.scala:264)
[38] org.apache.spark.rdd.MapPartitionsRDD.compute
(MapPartitionsRDD.scala:38)
[39] org.apache.spark.rdd.RDD.computeOrReadCheckpoint (RDD.scala:297)
[40] org.apache.spark.rdd.RDD.iterator (RDD.scala:264)
[41] org.apache.spark.rdd.MapPartitionsRDD.compute
(MapPartitionsRDD.scala:38)
[42] org.apache.spark.rdd.RDD.computeOrReadCheckpoint (RDD.scala:297)
[43] org.apache.spark.rdd.RDD.iterator (RDD.scala:264)
[44] org.apache.spark.rdd.MapPartitionsRDD.compute
(MapPartitionsRDD.scala:38)
[45] org.apache.spark.rdd.RDD.computeOrReadCheckpoint (RDD.scala:297)
[46] org.apache.spark.rdd.RDD.iterator (RDD.scala:264)
[47] org.apache.spark.scheduler.ShuffleMapTask.runTask
(ShuffleMapTask.scala:73)
[48] org.apache.spark.scheduler.ShuffleMapTask.runTask
(ShuffleMapTask.scala:41)
[49] org.apache.spark.scheduler.Task.run (Task.scala:88)
[50] org.apache.spark.executor.Executor$TaskRunner.run
(Executor.scala:214)
[51] java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1,142)
[52] java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:617)
[53] java.lang.Thread.run (Thread.java:745)
*In the mesos-slave where the task is hung:*
*lsof -p 27391 | grep amazon*
java 27391 matrix-data-pipes-api 155u IPv6 611285
0t0 TCP devu-saPerf3r-mesosslave-10-1-69-91:35665->s3-1-w.amazonaws.com:https
(ESTABLISHED)
*in gdb, info threads:*
12 Thread 0x7f09d5741700 (LWP 27493) "java" 0x00007f0a016827eb in
__libc_recv (fd=155, buf=0x7f09d572d6b0, n=5, flags=-1) at
../sysdeps/unix/sysv/linux/x86_64/recv.c:33
*in jdb:*
*print the request to S3:*
Executor task launch worker-3[20] dump request
request = {
resourcePath:
"devu/saPerf3r-bb6bf633-e03f-43d7-b049-1e0166762e4b/agent_smith_passthrough_performance12/v2/_full/part-r-00276-65a9fb14-28ac-4ac7-9901-c73f4754938d"
parameters: instance of java.util.HashMap(id=8031)
headers: instance of java.util.HashMap(id=8032)
endpoint: instance of java.net.URI(id=8033)
serviceName: "Amazon S3"
originalRequest: instance of
com.amazonaws.services.s3.model.GetObjectMetadataRequest(id=8035)
httpMethod: instance of com.amazonaws.http.HttpMethodName(id=8036)
content: null
timeOffset: 0
metrics: instance of com.amazonaws.util.AWSRequestMetrics(id=8023)
}
*Print the local variables in SSLSocketFactory.connectSocket:*
Executor task launch worker-3[11] locals
Method arguments:
connectTimeout = 50000
socket = instance of sun.security.ssl.SSLSocketImpl(id=8016)
host = instance of org.apache.http.HttpHost(id=8017)
remoteAddress = instance of
org.apache.http.conn.HttpInetSocketAddress(id=8018)
localAddress = null
context = null
Local variables:
sock = instance of sun.security.ssl.SSLSocketImpl(id=8016)
sslsock = instance of sun.security.ssl.SSLSocketImpl(id=8016)
I'm not able to print the local variable info
for SocketInputStream.socketRead in jdb because the jdk is not compiled
with -g.
Could you please advise what the problem might be? Please let me know if I
need to provide other information or run other experiment to help diagnose
the problem.
Thanks a lot in advanced!
Sa