You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Dyno (Jira)" <ji...@apache.org> on 2020/06/04 00:04:00 UTC
[jira] [Commented] (HADOOP-17063) S3ABlockOutputStream.putObject
looks stuck and never timeout
[ https://issues.apache.org/jira/browse/HADOOP-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125413#comment-17125413 ]
Dyno commented on HADOOP-17063:
-------------------------------
log in the spark executor,
Jun 3, 2020 @ 22:57:23.032
2020-06-03 22:57:23,032 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
Jun 3, 2020 @ 22:57:23.0322020-06-03 22:57:23,032 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...105
Jun 3, 2020 @ 22:57:23.0322020-06-03 22:57:23,032 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
Jun 3, 2020 @ 22:57:22.9732020-06-03 22:57:22,973 INFO util.ShutdownHookManager: Deleting directory /var/data/spark-ff208630-1fc7-48bc-93a1-6bdf94921c64/spark-eb4613f4-a41a-4985-845b-34b58ae95c50
Jun 3, 2020 @ 22:57:22.9722020-06-03 22:57:22,972 INFO util.ShutdownHookManager: Shutdown hook called
Jun 3, 2020 @ 22:57:22.964
2020-06-03 22:57:22,964 INFO storage.DiskBlockManager: Shutdown hook called
Jun 3, 2020 @ 22:57:22.960
2020-06-03 22:57:22,960 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM <-- kill to make it move forward
Jun 3, 2020 @ 21:52:40.5502020-06-03 21:52:40,549 INFO executor.Executor: Finished task 2927.0 in stage 12.0 (TID 31771). 4696 bytes result sent to driver
Jun 3, 2020 @ 21:52:40.5412020-06-03 21:52:40,541 INFO output.FileOutputCommitter: Saved output of task 'attempt_20200603213232_0012_m_002927_31771' to s3a://com....
Jun 3, 2020 @ 21:52:40.5412020-06-03 21:52:40,541 INFO mapred.SparkHadoopMapRedUtil: attempt_20200603213232_0012_m_002927_31771: Committed
Jun 3, 2020 @ 21:52:34.9722020-06-03 21:52:34,971 INFO executor.Executor: Finished task 2922.0 in stage 12.0 (TID 31766). 4696 bytes result sent to driver
Jun 3, 2020 @ 21:52:34.9632020-06-03 21:52:34,962 INFO output.FileOutputCommitter: Saved out ...
> S3ABlockOutputStream.putObject looks stuck and never timeout
> ------------------------------------------------------------
>
> Key: HADOOP-17063
> URL: https://issues.apache.org/jira/browse/HADOOP-17063
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.2.1
> Environment: hadoop 3.2.1
> spark 2.4.4
>
> Reporter: Dyno
> Priority: Major
>
> {code}
> sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:523) com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82) org.apache.hadoop.fs.s3a.S3ABlockOutputStream.putObject(S3ABlockOutputStream.java:446) org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:365) org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:64) org.apache.parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:685) org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:122) org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165) org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:42) org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:57) org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:74) org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247) org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242) org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394) org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248) org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170) org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169) org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) org.apache.spark.scheduler.Task.run(Task.scala:123) org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)
> {code}
>
> we are using spark 2.4.4 with hadoop 3.2.1 on kubernetes/spark-operator, sometimes we see this hang with the stacktrace above. it looks like the putObject never return, we have to kill the executor to make the job move forward.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org