You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@predictionio.apache.org by Krajcs Ádám <ad...@profession.hu> on 2018/07/25 07:39:13 UTC
UR template train doubled the events number
Hi everybody,
We use predictionIO 0.12.1 and UR template 0.7.2. During the train the events number doubled and the train failed. I’ve never seen anything like this. What happened?
Here are the event numbers during the train:
2018-07-23 18:40:01 - pio_event:events_1 - 9885028
2018-07-23 18:45:01 - pio_event:events_1 - 9886805
2018-07-23 18:50:01 - pio_event:events_1 - 11553133
2018-07-23 18:55:01 - pio_event:events_1 - 17096412
2018-07-23 19:00:01 - pio_event:events_1 – 17618479
And the output of the train:
Tue Jul 24 18:30:20 CEST 2018
Start -== Profession ==- PIO train...
=== SPARK Gui: ===>
[INFO] [Runner$] Submission command: /opt/PredictionIO-0.12.1/vendors/spark/bin/spark-submit --driver-memory 6G --master spark://xxx.xxx.hu:7077 --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/home/pio/UR/profession-job/target/scala-2.11/universal-recommender_2.11-0.7.2.jar,file:/home/pio/UR/profession-job/target/scala-2.11/universal-recommender-assembly-0.7.2-deps.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-hdfs-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-s3-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-jdbc-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-hbase-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-elasticsearch-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-localfs-assembly-0.12.1.jar --files file:/opt/PredictionIO-0.12.1/conf/log4j.properties,file:/opt/PredictionIO-0.12.1/vendors/elasticsearch/config/elasticsearch.yml,file:/
opt/PredictionIO-0.12.1/vendors/hbase/conf/hbase-site.xml --driver-class-path /opt/PredictionIO-0.12.1/conf:/opt/PredictionIO-0.12.1/vendors/elasticsearch/config:/opt/PredictionIO-0.12.1/vendors/hbase/conf --driver-java-options -Dpio.log.dir=/var/log/predictionio/ file:/opt/PredictionIO-0.12.1/lib/pio-assembly-0.12.1.jar --engine-id com.actionml.RecommendationEngine --engine-version 7c5332885ce2f37d0f45f8068f57af06eb76b290 --engine-variant file:/home/pio/UR/profession-job/engine.json --verbosity 0 --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/pio/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=xxx.xxx.hu,PIO_STORAGE_SOURCES_HBASE_HOME=/opt/PredictionIO-0.12.1/vendors/hbase,PIO_HOME=/opt/PredictionIO-0.12.1,PIO_FS_ENGINESDIR=/home/pio/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/home/pio/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_S
TORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=/opt/PredictionIO-0.12.1/vendors/elasticsearch,PIO_FS_TMPDIR=/home/pio/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_LOG_DIR=/var/log/predictionio/,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/opt/PredictionIO-0.12.1/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
[INFO] [RecommendationEngine$]
_ _ __ __ _
/\ | | (_) | \/ | |
/ \ ___| |_ _ ___ _ __ | \ / | |
/ /\ \ / __| __| |/ _ \| '_ \| |\/| | |
/ ____ \ (__| |_| | (_) | | | | | | | |____
/_/ \_\___|\__|_|\___/|_| |_|_| |_|______|
[INFO] [Engine] Extracting datasource params...
[INFO] [WorkflowUtils$] No 'name' is found. Default empty String will be used.
[INFO] [Engine] Datasource params: (,DataSourceParams(professionJob201806,List(view, jobapply),Some(EventWindow(Some(26 days),true,false)),None))
[INFO] [Engine] Extracting preparator params...
[INFO] [Engine] Preparator params: (,Empty)
[INFO] [Engine] Extracting serving params...
[INFO] [Engine] Serving params: (,Empty)
[INFO] [log] Logging initialized @6782ms
[INFO] [Server] jetty-9.2.z-SNAPSHOT
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@51a651c1{/jobs,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@173f1614{/jobs/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@6c184d4d{/jobs/job,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@7645f03e{/jobs/job/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@158e9f6e{/stages,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@7a6ea47d{/stages/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@54b2fc58{/stages/stage,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@daf22f0{/stages/stage/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@6cbbb9c4{/stages/pool,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@63f9b562{/stages/pool/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@20ead579{/storage,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5426cb36{/storage/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@65e22def{/storage/rdd,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@4b1ec694{/storage/rdd/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@3d299393{/environment,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5f34907b{/environment/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2b4d4327{/executors,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@16da1abc{/executors/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@671ea6ff{/executors/threadDump,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@1c52552f{/executors/threadDump/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5dc769f9{/static,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@1b0e9707{/,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@9b5f3c7{/api,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@74024f3{/jobs/job/kill,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@61ae0d43{/stages/stage/kill,null,AVAILABLE,@Spark}
[INFO] [ServerConnector] Started Spark@5820070c{HTTP/1.1}{0.0.0.0:4040}
[INFO] [Server] Started @7032ms
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@245253d8{/metrics/json,null,AVAILABLE,@Spark}
[INFO] [DataSource]
╔════════════════════════════════════════════════════════════╗
║ Init DataSource ║
║ ══════════════════════════════════════════════════════════ ║
║ App name professionJob201806 ║
║ Event window Some(EventWindow(Some(26 days),true,false)) ║
║ Event names List(view, jobapply) ║
║ Min events per user None ║
╚════════════════════════════════════════════════════════════╝
[INFO] [URAlgorithm]
╔════════════════════════════════════════════════════════════╗
║ Init URAlgorithm ║
║ ══════════════════════════════════════════════════════════ ║
║ App name professionJob201806 ║
║ ES index name urindexsecondary ║
║ ES type name items ║
║ RecsModel all ║
║ Event names List(view, jobapply) ║
║ ══════════════════════════════════════════════════════════ ║
║ Random seed -853494969 ║
║ MaxCorrelatorsPerEventType 50 ║
║ MaxEventsPerEventType 500 ║
║ BlacklistEvents List(jobapply) ║
║ ══════════════════════════════════════════════════════════ ║
║ User bias 1.0 ║
║ Item bias 1.0 ║
║ Max query events 40 ║
║ Limit 20 ║
║ ══════════════════════════════════════════════════════════ ║
║ Rankings: ║
║ popular Some(popRank) ║
╚════════════════════════════════════════════════════════════╝
[INFO] [Engine$] EngineWorkflow.train
[INFO] [Engine$] DataSource: com.actionml.DataSource@5b35c7d
[INFO] [Engine$] Preparator: com.actionml.Preparator@56826a75
[INFO] [Engine$] AlgorithmList: List(com.actionml.URAlgorithm@14d25b6e)
[INFO] [Engine$] Data sanity check is on.
[Stage 0:> (0 + 3) / 3]
[Stage 0:> (0 + 3) / 3]
[Stage 0:===================> (1 + 2) / 3]
[Stage 0:=======================================> (2 + 1) / 3]
[Stage 0:=======================================> (2 + 1) / 3]
[Stage 1:> (0 + 3) / 3]
[Stage 1:> (0 + 3) / 3]
[Stage 1:===================> (1 + 2) / 3]
[Stage 1:=======================================> (2 + 1) / 3]
[Stage 1:=======================================> (2 + 1) / 3]
[Stage 2:> (0 + 3) / 3]
[Stage 2:>
(0 + 3) / 3]
[Stage 2:> (0 + 3) / 3]
[Stage 2:> (0 + 3) / 3]
[Stage 2:> (0 + 3) / 3]
[Stage 2:> (0 + 3) / 3]
[Stage 2:===================> (1 + 2) / 3]
[Stage 2:=======================================> (2 + 1) / 3]
[Stage 3:> (0 + 3) / 3][Stage 5:> (0 + 3) / 3]
[Stage 3:> (0 + 3) / 3][Stage 5:> (0 + 3) / 3]
[Stage 3:> (0 + 3) / 3][Stage 5:> (0 + 3) / 3]
[Stage 3:> (0 + 3) / 3][Stage 5:> (0 + 3) / 3]
[Stage 3:> (0 + 3) / 3][Stage 5
:> (0 + 3) / 3]
[Stage 3:> (0 + 3) / 3][Stage 5:======> (1 + 2) / 3]
[Stage 3:======> (1 + 2) / 3][Stage 5:======> (1 + 2) / 3]
[Stage 3:============> (2 + 1) / 3][Stage 5:======> (1 + 2) / 3]
[Stage 3:============> (2 + 1) / 3][Stage 5:============> (2 + 1) / 3]
[Stage 3:============> (2 + 1) / 3][Stage 6:> (0 + 0) / 3]
[Stage 3:=======================================> (2 + 1) / 3]
[Stage 3:============> (2 + 1) / 3][Stage 6:> (0 + 3) / 3]
[Stage 3:============> (2 + 1) / 3][Stage 6:> (0 + 3) / 3]
[Stage 3:============> (2 + 1) / 3][Stage 6:> (0 + 3) / 3]
[Stage 6:> (0 + 3) / 3]
[Stage 6:> (0 + 3) / 3]
[Stage 6:===================> (1 +
2) / 3]
[Stage 6:=======================================> (2 + 1) / 3]
[Stage 7:> (0 + 8) / 8]
[Stage 7:> (0 + 8) / 8]
[Stage 7:> (0 + 8) / 8]
[Stage 7:> (0 + 8) / 8]
[Stage 7:> (0 + 8) / 8]
[Stage 7:> (0 + 8) / 8]
[Stage 7:> (0 + 8) / 8]
[Stage 7:=======> (1 + 7) / 8]
[Stage 7:==============> (2 + 6) / 8]
[Stage 7:======================> (3 + 5) / 8]
[Stage 7:=============================> (4 + 4) / 8]
[Stage 7:========
============================> (5 + 3) / 8]
[Stage 7:============================================> (6 + 2) / 8]
[Stage 8:> (0 + 8) / 8]
[Stage 8:> (0 + 8) / 8]
[Stage 8:> (0 + 8) / 8]
[Stage 8:> (0 + 8) / 8]
[Stage 8:=======> (1 + 7) / 8]
[Stage 8:==============> (2 + 6) / 8]
[Stage 8:======================> (3 + 5) / 8]
[Stage 8:=============================> (4 + 4) / 8]
[Stage 8:====================================> (5 + 3) / 8]
[Stage 8:===================================================> (7 + 1) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:==> (1 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:======> (3 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:=========> (4 + 4) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:===========> (5 + 3) / 8]
[Stage 9:======> (1 + 2) / 3][Stage 13:===========> (5 + 3) / 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:===========>
(5 + 3) / 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:===========> (5 + 3) / 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:===========> (5 + 3) / 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:=============> (6 + 2) / 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:===============> (7 + 1) / 8]
[Stage 9:=======================================> (2 + 1) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:
> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [HeartbeatReceiver] Removing executor 0 with no recent heartbeats: 172452 ms exceeds timeout 120000 ms
[ERROR] [TaskSchedulerImpl] Lost executor 0 on piotrain1.profession.hu: Executor heartbeat timed out after 172452 ms
[WARN] [TaskSetManager] Lost task 2.0 in stage 14.0 (TID 47, piotrain1.profession.hu, executor 0): ExecutorLostFailure (executor 0 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 172452 ms
[WARN] [TaskSetManager] Lost task 1.0 in stage 14.0 (TID 46, piotrain1.profession.hu, executor 0): ExecutorLostFailure (executor 0 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 172452 ms
[WARN] [TaskSetManager] Lost task 0.0 in stage 14.0 (TID 45, piotrain1.profession.hu, executor 0): ExecutorLostFailure (executor 0 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 172452 ms
[WARN] [TransportChannelHandler] Exception in connection from /172.31.23.58:34288
[ERROR] [TaskSchedulerImpl] Lost executor 0 on piotrain1.profession.hu: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 2.1 in stage 14.0 (TID 50, piotrain1.profession.hu, executor 0): ExecutorLostFailure (executor 0 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 1.1 in stage 14.0 (TID 49, piotrain1.profession.hu, executor 0): ExecutorLostFailure (executor 0 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 0.1 in stage 14.0 (TID 48, piotrain1.profession.hu, executor 0): ExecutorLostFailure (executor 0 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[Stage 14:> (0 + 0) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [TaskSetManager] Lost task 0.2 in stage 14.0 (TID 51, piotrain1.profession.hu, executor 125): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=0, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 1.2 in stage 14.0 (TID 52, piotrain1.profession.hu, executor 125): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=1, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 2.2 in stage 14.0 (TID 53, piotrain1.profession.hu, executor 125): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=2, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:======> (1 + 2) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:======> (1 + 2) / 3][Stage 10:======> (1 + 2) / 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1) / 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1) / 3]
[Stage 10:======================================> (2 + 1) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:>
(0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:===================> (1 + 2) / 3]
[Stage 11:======================================> (2 + 1) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:===================>
(1 + 2) / 3]
[Stage 12:======================================> (2 + 1) / 3]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:=======> (1 + 7) / 8]
[Stage 13:==============> (2 + 6) / 8]
[Stage 13:=====================> (3 + 5)
/ 8]
[Stage 13:=============================> (4 + 4) / 8]
[Stage 13:====================================> (5 + 3) / 8]
[Stage 13:===========================================> (6 + 2) / 8]
[Stage 13:==================================================> (7 + 1) / 8]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [HeartbeatReceiver] Removing executor 125 with no recent heartbeats: 177831 ms exceeds timeout 120000 ms
[ERROR] [TaskSchedulerImpl] Lost executor 125 on piotrain1.profession.hu: Executor heartbeat timed out after 177831 ms
[WARN] [TaskSetManager] Lost task 0.0 in stage 14.1 (TID 74, piotrain1.profession.hu, executor 125): ExecutorLostFailure (executor 125 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 177831 ms
[WARN] [TaskSetManager] Lost task 2.0 in stage 14.1 (TID 76, piotrain1.profession.hu, executor 125): ExecutorLostFailure (executor 125 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 177831 ms
[WARN] [TaskSetManager] Lost task 1.0 in stage 14.1 (TID 75, piotrain1.profession.hu, executor 125): ExecutorLostFailure (executor 125 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 177831 ms
[WARN] [TransportChannelHandler] Exception in connection from /172.31.23.58:34448
[ERROR] [TaskSchedulerImpl] Lost executor 125 on piotrain1.profession.hu: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 1.1 in stage 14.1 (TID 77, piotrain1.profession.hu, executor 125): ExecutorLostFailure (executor 125 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 0.1 in stage 14.1 (TID 79, piotrain1.profession.hu, executor 125): ExecutorLostFailure (executor 125 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 2.1 in stage 14.1 (TID 78, piotrain1.profession.hu, executor 125): ExecutorLostFailure (executor 125 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[Stage 14:> (0 + 0) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [TaskSetManager] Lost task 1.2 in stage 14.1 (TID 82, piotrain1.profession.hu, executor 257): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=1, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 0.2 in stage 14.1 (TID 81, piotrain1.profession.hu, executor 257): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=0, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 2.2 in stage 14.1 (TID 80, piotrain1.profession.hu, executor 257): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=2, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:> (0 + 3) / 3][Stage 10:======> (1 + 2) / 3]
[Stage 9:======> (1 + 2) / 3][Stage 10:======> (1 + 2) / 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1) / 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:>
(0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:===================> (1 + 2) / 3]
[Stage 11:======================================> (2 + 1) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:===================> (1 + 2) / 3]
[Stage 12:======================================> (2 + 1) / 3]
[Stage 13:>
(0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:=======> (1 + 7) / 8]
[Stage 13:==============> (2 + 6) / 8]
[Stage 13:=====================> (3 + 5) / 8]
[Stage 13:=============================> (4 + 4) / 8]
[Stage 13:====================================> (5 + 3)
/ 8]
[Stage 13:===========================================> (6 + 2) / 8]
[Stage 13:==================================================> (7 + 1) / 8]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [HeartbeatReceiver] Removing executor 257 with no recent heartbeats: 146964 ms exceeds timeout 120000 ms
[ERROR] [TaskSchedulerImpl] Lost executor 257 on piotrain1.profession.hu: Executor heartbeat timed out after 146964 ms
[WARN] [TaskSetManager] Lost task 1.0 in stage 14.2 (TID 104, piotrain1.profession.hu, executor 257): ExecutorLostFailure (executor 257 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 146964 ms
[WARN] [TaskSetManager] Lost task 0.0 in stage 14.2 (TID 103, piotrain1.profession.hu, executor 257): ExecutorLostFailure (executor 257 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 146964 ms
[WARN] [TaskSetManager] Lost task 2.0 in stage 14.2 (TID 105, piotrain1.profession.hu, executor 257): ExecutorLostFailure (executor 257 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 146964 ms
[Stage 14:> (0 + 3) / 3][WARN] [TransportChannelHandler] Exception in connection from /172.31.23.58:34522
[ERROR] [TaskSchedulerImpl] Lost executor 257 on piotrain1.profession.hu: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 0.1 in stage 14.2 (TID 107, piotrain1.profession.hu, executor 257): ExecutorLostFailure (executor 257 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 2.1 in stage 14.2 (TID 106, piotrain1.profession.hu, executor 257): ExecutorLostFailure (executor 257 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 1.1 in stage 14.2 (TID 108, piotrain1.profession.hu, executor 257): ExecutorLostFailure (executor 257 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[Stage 14:> (0 + 0) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [TaskSetManager] Lost task 1.2 in stage 14.2 (TID 109, piotrain1.profession.hu, executor 379): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=1, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 0.2 in stage 14.2 (TID 111, piotrain1.profession.hu, executor 379): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=0, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 2.2 in stage 14.2 (TID 110, piotrain1.profession.hu, executor 379): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=2, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:======> (1 + 2) / 3][Stage 10:======> (1 + 2) / 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:======> (1 + 2) / 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1) / 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1) / 3]
[Stage 9:=======================================> (2 + 1) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:>
(0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:===================> (1 + 2) / 3]
[Stage 11:======================================> (2 + 1) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:===================>
(1 + 2) / 3]
[Stage 12:======================================> (2 + 1) / 3]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:=======> (1 + 7) / 8]
[Stage 13:==============> (2 + 6) / 8]
[Stage 13:=============================> (4 + 4)
/ 8]
[Stage 13:====================================> (5 + 3) / 8]
[Stage 13:===========================================> (6 + 2) / 8]
[Stage 13:==================================================> (7 + 1) / 8]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [HeartbeatReceiver] Removing executor 379 with no recent heartbeats: 164583 ms exceeds timeout 120000 ms
[ERROR] [TaskSchedulerImpl] Lost executor 379 on piotrain1.profession.hu: Executor heartbeat timed out after 164583 ms
[WARN] [TaskSetManager] Lost task 2.0 in stage 14.3 (TID 134, piotrain1.profession.hu, executor 379): ExecutorLostFailure (executor 379 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164583 ms
[WARN] [TaskSetManager] Lost task 1.0 in stage 14.3 (TID 133, piotrain1.profession.hu, executor 379): ExecutorLostFailure (executor 379 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164583 ms
[WARN] [TaskSetManager] Lost task 0.0 in stage 14.3 (TID 132, piotrain1.profession.hu, executor 379): ExecutorLostFailure (executor 379 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164583 ms
[WARN] [TransportChannelHandler] Exception in connection from /172.31.23.58:34594
[ERROR] [TaskSchedulerImpl] Lost executor 379 on piotrain1.profession.hu: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 2.1 in stage 14.3 (TID 137, piotrain1.profession.hu, executor 379): ExecutorLostFailure (executor 379 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 1.1 in stage 14.3 (TID 136, piotrain1.profession.hu, executor 379): ExecutorLostFailure (executor 379 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 0.1 in stage 14.3 (TID 135, piotrain1.profession.hu, executor 379): ExecutorLostFailure (executor 379 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[Stage 14:> (0 + 0) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [TaskSetManager] Lost task 2.2 in stage 14.3 (TID 140, piotrain1.profession.hu, executor 509): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=2, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 1.2 in stage 14.3 (TID 139, piotrain1.profession.hu, executor 509): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=1, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 0.2 in stage 14.3 (TID 138, piotrain1.profession.hu, executor 509): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=0, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[INFO] [ServerConnector] Stopped Spark@5820070c{HTTP/1.1}{0.0.0.0:4040}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@61ae0d43{/stages/stage/kill,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@74024f3{/jobs/job/kill,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@9b5f3c7{/api,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@1b0e9707{/,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@5dc769f9{/static,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@1c52552f{/executors/threadDump/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@671ea6ff{/executors/threadDump,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@16da1abc{/executors/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@2b4d4327{/executors,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@5f34907b{/environment/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@3d299393{/environment,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@4b1ec694{/storage/rdd/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@65e22def{/storage/rdd,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@5426cb36{/storage/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@20ead579{/storage,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@63f9b562{/stages/pool/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@6cbbb9c4{/stages/pool,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@daf22f0{/stages/stage/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@54b2fc58{/stages/stage,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@7a6ea47d{/stages/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@158e9f6e{/stages,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@7645f03e{/jobs/job/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@6c184d4d{/jobs/job,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@173f1614{/jobs/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@51a651c1{/jobs,null,UNAVAILABLE,@Spark}
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: ResultStage 14 (foreachPartition at HBPEvents.scala:117) has failed the maximum allowable number of times: 4. Most recent failure reason: org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1262)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1925)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1938)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1951)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1965)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:926)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:924)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:924)
at org.apache.predictionio.data.storage.hbase.HBPEvents.delete(HBPEvents.scala:117)
at org.apache.predictionio.core.SelfCleaningDataSource$class.removePEvents(SelfCleaningDataSource.scala:198)
at com.actionml.DataSource.removePEvents(DataSource.scala:48)
at org.apache.predictionio.core.SelfCleaningDataSource$class.wipePEvents(SelfCleaningDataSource.scala:184)
at com.actionml.DataSource.wipePEvents(DataSource.scala:48)
at org.apache.predictionio.core.SelfCleaningDataSource$class.cleanPersistedPEvents(SelfCleaningDataSource.scala:170)
at com.actionml.DataSource.cleanPersistedPEvents(DataSource.scala:48)
at com.actionml.DataSource.readTraining(DataSource.scala:70)
at com.actionml.DataSource.readTraining(DataSource.scala:48)
at org.apache.predictionio.controller.PDataSource.readTrainingBase(PDataSource.scala:40)
at org.apache.predictionio.controller.Engine$.train(Engine.scala:642)
at org.apache.predictionio.controller.Engine.train(Engine.scala:176)
at org.apache.predictionio.workflow.CoreWorkflow$.runTrain(CoreWorkflow.scala:67)
at org.apache.predictionio.workflow.CreateWorkflow$.main(CreateWorkflow.scala:251)
at org.apache.predictionio.workflow.CreateWorkflow.main(CreateWorkflow.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
real 138m25.051s
user 0m45.616s
sys 0m5.844s
PIO -== Profession ==- train done.
Tue Jul 24 20:48:45 CEST 2018
Best regards,
Adam Krajcs
RE: UR template train doubled the events number
Posted by Krajcs Ádám <ad...@profession.hu>.
I’ve tried to build the db-cleaner template with pio (0.12.1):
{
"id": "professionJob201806",
"description": "Default settings",
"engineFactory": "com.actionml.templates.dbcleaner.DBCleaner",
"datasource": {
"params" : {
"appName": "professionJob201806",
"eventWindow": {
"duration": "28 days",
"removeDuplicates": true,
"compressProperties": true
}
}
},
"algorithms": [
{
"name": "db-cleaner-algo",
"params": {
"appName": "professionJob201806"
}
}
]
}
[ERROR] [Engine$] No engine found. Your build might have failed. Aborting.
Here:
https://github.com/actionml/db-cleaner
"engineFactory": "com.actionml.DBClean",
In the code:
"engineFactory": "com.actionml.templates.dbcleaner.DBCleaner"
I’ve tried both, buti t doesn’t work.
Regards,
Adam Krajcs
From: Pat Ferrel <pa...@occamsmachete.com>
Sent: Wednesday, July 25, 2018 5:54 PM
To: Krajcs Ádám <ad...@profession.hu>; user@predictionio.apache.org
Subject: Re: UR template train doubled the events number
Don’t use the DB cleaning feature inside the template’s engine.json. It takes too much memory and time to do for each `pio train` we have a template that only does cleaning: https://github.com/actionml/db-cleaner with docs here: http://actionml.com/docs/db_cleaner_template
The reason the events doubled is probably because the clean creates them before dropping the unlceaned ones but dies due to lack of memory.
From: Krajcs Ádám <ad...@profession.hu>
Reply: user@predictionio.apache.org<ma...@predictionio.apache.org> <us...@predictionio.apache.org>
Date: July 25, 2018 at 12:39:13 AM
To: user@predictionio.apache.org<ma...@predictionio.apache.org> <us...@predictionio.apache.org>
Subject: UR template train doubled the events number
Hi everybody,
We use predictionIO 0.12.1 and UR template 0.7.2. During the train the events number doubled and the train fa
iled. I’ve never seen anything like this. What happened?
Here are the event numbers during the train:
2018-07-23 18:40:01 - pio_event:events_1 - 9885028
2018-07-23 18:45:01 - pio_event:events_1 - 9886805
2018-07-23 18:50:01 - pio_event:events_1 - 11553133
2018-07-23 18:55:01 - pio_event:events_1 - 17096412
2018-07-23 19:00:01 - pio_event:events_1 – 17618479
And the output of the train:
Tue Jul 24 18:30:20 CEST 2018
Start -== Profession ==- PIO train...
=== SPARK Gui: ===>
[INFO] [Runner$] Submission command: /opt/PredictionIO-0.12.1/vendors/spark/bin/spark-submit --driver-memory 6G --master spark://xxx.xxx.hu:7077<http://xxx.xxx.hu:7077> --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/home/pio/UR/profession-job/target/scala-2.11/universal-recommender_2.11-0.7.2.jar,file:/home/pio/UR/profession-job/target/scala-2.11/universal-recommender-assembly-0.7.2-deps.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-hdfs-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-s3-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-jdbc-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-hbase-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-elasticsearch-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-localfs-assembly-0.12.1.jar --files file:/opt/PredictionIO-0.12.1/conf/log4j.properties,file:/opt/PredictionIO-0.12.1/vendors/elasticsearch/config/elasticsearch.yml,file:/
opt/PredictionIO-0.12.1/vendors/hbase/conf/hbase-site.xml --driver-class-path /opt/PredictionIO-0.12.1/conf:/opt/PredictionIO-0.12.1/vendors/elasticsearch/config:/opt/PredictionIO-0.12.1/vendors/hbase/conf --driver-java-options -Dpio.log.dir=/var/log/predictionio/ file:/opt/PredictionIO-0.12.1/lib/pio-assembly-0.12.1.jar --engine-id com.actionml.RecommendationEngine --engine-version 7c5332885ce2f37d0f45f8068f57af06eb76b290 --engine-variant file:/home/pio/UR/profession-job/engine.json --verbosity 0 --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/pio/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=xxx.xxx.hu<http://xxx.xxx.hu>,PIO_STORAGE_SOURCES_HBASE_HOME=/opt/PredictionIO-0.12.1/vendors/hbase,PIO_HOME=/opt/PredictionIO-0.12.1,PIO_FS_ENGINESDIR=/home/pio/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/home/pio/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_S
TORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=/opt/PredictionIO-0.12.1/vendors/elasticsearch,PIO_FS_TMPDIR=/home/pio/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_LOG_DIR=/var/log/predictionio/,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/opt/PredictionIO-0.12.1/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
[INFO] [RecommendationEngine$]
_ _ __ __ _
/\ | | (_) | \/ | |
/ \ ___| |_ _ ___ _ __ | \ / | |
/ /\ \ / __| __| |/ _ \| '_ \| |\/| | |
/ ____ \ (__| |_| | (_) | | | | | | | |____
/_/ \_\___|\__|_|\___/|_| |_|_| |_|______|
[INFO] [Engine] Extracting datasource params...
[INFO] [WorkflowUtils$] No 'name' is found. Default empty String will be used.
[INFO] [Engine] Datasource params: (,DataSourceParams(professionJob201806,List(view, jobapply),Some(EventWindow(Some(26 days),true,false)),None))
[INFO] [Engine] Extracting preparator params...
[INFO] [Engine] Preparator params: (,Empty)
[INFO] [Engine] Extracting serving params...
[INFO] [Engine] Serving params: (,Empty)
[INFO] [log] Logging initialized @6782ms
[INFO] [Server] jetty-9.2.z-SNAPSHOT
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@51a651c1{/jobs,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@51a651c1%7b/jobs,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@173f1614{/jobs/json,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@173f1614%7b/jobs/json,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@6c184d4d{/jobs/job,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@6c184d4d%7b/jobs/job,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@7645f03e{/jobs/job/json,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@7645f03e%7b/jobs/job/json,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@158e9f6e{/stages,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@158e9f6e%7b/stages,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@7a6ea47d{/stages/json,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@7a6ea47d%7b/stages/json,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@54b2fc58{/stages/stage,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@54b2fc58%7b/stages/stage,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@daf22f0{/stages/stage/json,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@daf22f0%7b/stages/stage/json,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@6cbbb9c4{/stages/pool,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@6cbbb9c4%7b/stages/pool,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@63f9b562{/stages/pool/json,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@63f9b562%7b/stages/pool/json,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@20ead579{/storage,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@20ead579%7b/storage,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5426cb36{/storage/json,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@5426cb36%7b/storage/json,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@65e22def{/storage/rdd,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@65e22def%7b/storage/rdd,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@4b1ec694{/storage/rdd/json,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@4b1ec694%7b/storage/rdd/json,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@3d299393{/environment,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@3d299393%7b/environment,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5f34907b{/environment/json,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@5f34907b%7b/environment/json,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2b4d4327{/executors,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@2b4d4327%7b/executors,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@16da1abc{/executors/json,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@16da1abc%7b/executors/json,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@671ea6ff{/executors/threadDump,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@671ea6ff%7b/executors/threadDump,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@1c52552f{/executors/threadDump/json,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@1c52552f%7b/executors/threadDump/json,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5dc769f9{/static,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@5dc769f9%7b/static,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@1b0e9707{/,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@1b0e9707%7b/,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@9b5f3c7{/api,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@9b5f3c7%7b/api,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@74024f3{/jobs/job/kill,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@74024f3%7b/jobs/job/kill,null,AVAILABLE,@Spark%7d>
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@61ae0d43{/stages/stage/kill,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@61ae0d43%7b/stages/stage/kill,null,AVAILABLE,@Spark%7d>
[INFO] [ServerConnector] Started Spark@5820070c{HTTP/1.1}{0.0.0.0:4040<http://0.0.0.0:4040>}
[INFO] [Server] Started @7032ms
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@245253d8{/metrics/json,null,AVAILABLE,@Spark}<mailto:o.s.j.s.ServletContextHandler@245253d8%7b/metrics/json,null,AVAILABLE,@Spark%7d>
[INFO] [DataSource]
╔════════════════════════════════════════════════════════════╗
║ Init DataSource ║
║ ══════════════════════════════════════════════════════════ ║
║ App name professionJob201806 ║
║ Event window Some(EventWindow(Some(26 days),true,false)) ║
║ Event names List(view, jobapply) ║
║ Min events per user None ║
╚════════════════════════════════════════════════════════════╝
[INFO] [URAlgorithm]
╔════════════════════════════════════════════════════════════╗
║ Init URAlgorithm ║
║ ══════════════════════════════════════════════════════════ ║
║ App name professionJob201806 ║
║ ES index name urindexsecondary ║
║ ES type name items ║
║ RecsModel all ║
║ Event names List(view, jobapply) ║
║ ══════════════════════════════════════════════════════════ ║
║ Random seed -853494969 ║
║ MaxCorrelatorsPerEventType 50 ║
║ MaxEventsPerEventType 500 ║
║ BlacklistEvents List(jobapply) ║
║ ══════════════════════════════════════════════════════════ ║
║ User bias 1.0 ║
║ Item bias 1.0 ║
║ Max query events 40 ║
║ Limit 20 ║
║ ══════════════════════════════════════════════════════════ ║
║ Rankings: ║
║ popular Some(popRank) ║
╚════════════════════════════════════════════════════════════╝
[INFO] [Engine$] EngineWorkflow.train
[INFO] [Engine$] DataSource: com.actionml.DataSource@5b35c7d<ma...@5b35c7d>
[INFO] [Engine$] Preparator: com.actionml.Preparator@56826a75<ma...@56826a75>
[INFO] [Engine$] AlgorithmList: List(com.actionml.URAlgorithm@14d25b6e<ma...@14d25b6e>)
[INFO] [Engine$] Data sanity check is on.
[Stage 0:> (0 + 3) / 3]
[Stage 0:> (0 + 3) / 3]
[Stage 0:===================> (1 + 2) / 3]
[Stage 0:=======================================> (2 + 1) / 3]
[Stage 0:=======================================> (2 + 1) / 3]
[Stage 1:> (0 + 3) / 3]
[Stage 1:> (0 + 3) / 3]
[Stage 1:===================> (1 + 2) / 3]
[Stage 1:=======================================> (2 + 1) / 3]
[Stage 1:=======================================> (2 + 1) / 3]
[Stage 2:> (0 + 3) / 3]
[Stage 2:>
(0 + 3) / 3]
[Stage 2:> (0 + 3) / 3]
[Stage 2:> (0 + 3) / 3]
[Stage 2:> (0 + 3) / 3]
[Stage 2:> (0 + 3) / 3]
[Stage 2:===================> (1 + 2) / 3]
[Stage 2:=======================================> (2 + 1) / 3]
[Stage 3:> (0 + 3) / 3][Stage 5:> (0 + 3) / 3]
[Stage 3:> (0 + 3) / 3][Stage 5:> (0 + 3) / 3]
[Stage 3:> (0 + 3) / 3][Stage 5:> (0 + 3) / 3]
[Stage 3:> (0 + 3) / 3][Stage 5:> (0 + 3) / 3]
[Stage 3:> (0 + 3) / 3][Stage 5
:> (0 + 3) / 3]
[Stage 3:> (0 + 3) / 3][Stage 5:======> (1 + 2) / 3]
[Stage 3:======> (1 + 2) / 3][Stage 5:======> (1 + 2) / 3]
[Stage 3:============> (2 + 1) / 3][Stage 5:======> (1 + 2) / 3]
[Stage 3:============> (2 + 1) / 3][Stage 5:============> (2 + 1) / 3]
[Stage 3:============> (2 + 1) / 3][Stage 6:> (0 + 0) / 3]
[Stage 3:=======================================> (2 + 1) / 3]
[Stage 3:============> (2 + 1) / 3][Stage 6:> (0 + 3) / 3]
[Stage 3:============> (2 + 1) / 3][Stage 6:> (0 + 3) / 3]
[Stage 3:============> (2 + 1) / 3][Stage 6:> (0 + 3) / 3]
[Stage 6:> (0 + 3) / 3]
[Stage 6:> (0 + 3) / 3]
[Stage 6:===================> (1 +
2) / 3]
[Stage 6:=======================================> (2 + 1) / 3]
[Stage 7:> (0 + 8) / 8]
[Stage 7:> (0 + 8) / 8]
[Stage 7:> (0 + 8) / 8]
[Stage 7:> (0 + 8) / 8]
[Stage 7:> (0 + 8) / 8]
[Stage 7:> (0 + 8) / 8]
[Stage 7:> (0 + 8) / 8]
[Stage 7:=======> (1 + 7) / 8]
[Stage 7:==============> (2 + 6) / 8]
[Stage 7:======================> (3 + 5) / 8]
[Stage 7:=============================> (4 + 4) / 8]
[Stage 7:========
============================> (5 + 3) / 8]
[Stage 7:============================================> (6 + 2) / 8]
[Stage 8:> (0 + 8) / 8]
[Stage 8:> (0 + 8) / 8]
[Stage 8:> (0 + 8) / 8]
[Stage 8:> (0 + 8) / 8]
[Stage 8:=======> (1 + 7) / 8]
[Stage 8:==============> (2 + 6) / 8]
[Stage 8:======================> (3 + 5) / 8]
[Stage 8:=============================> (4 + 4) / 8]
[Stage 8:====================================> (5 + 3) / 8]
[Stage 8:===================================================> (7 + 1) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:==> (1 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:======> (3 + 5) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:=========> (4 + 4) / 8]
[Stage 9:> (0 + 3) / 3][Stage 13:===========> (5 + 3) / 8]
[Stage 9:======> (1 + 2) / 3][Stage 13:===========> (5 + 3) / 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:===========>
(5 + 3) / 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:===========> (5 + 3) / 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:===========> (5 + 3) / 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:=============> (6 + 2) / 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:===============> (7 + 1) / 8]
[Stage 9:=======================================> (2 + 1) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:
> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [HeartbeatReceiver] Removing executor 0 with no recent heartbeats: 172452 ms exceeds timeout 120000 ms
[ERROR] [TaskSchedulerImpl] Lost executor 0 on piotrain1.profession.hu<http://piotrain1.profession.hu>: Executor heartbeat timed out after 172452 ms
[WARN] [TaskSetManager] Lost task 2.0 in stage 14.0 (TID 47, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 0): ExecutorLostFailure (executor 0 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 172452 ms
[WARN] [TaskSetManager] Lost task 1.0 in stage 14.0 (TID 46, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 0): ExecutorLostFailure (executor 0 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 172452 ms
[WARN] [TaskSetManager] Lost task 0.0 in stage 14.0 (TID 45, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 0): ExecutorLostFailure (executor 0 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 172452 ms
[WARN] [TransportChannelHandler] Exception in connection from /172.31.23.58:34288<http://172.31.23.58:34288>
[ERROR] [TaskSchedulerImpl] Lost executor 0 on piotrain1.profession.hu<http://piotrain1.profession.hu>: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 2.1 in stage 14.0 (TID 50, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 0): ExecutorLostFailure (executor 0 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 1.1 in stage 14.0 (TID 49, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 0): ExecutorLostFailure (executor 0 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 0.1 in stage 14.0 (TID 48, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 0): ExecutorLostFailure (executor 0 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[Stage 14:> (0 + 0) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [TaskSetManager] Lost task 0.2 in stage 14.0 (TID 51, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 125): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=0, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 1.2 in stage 14.0 (TID 52, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 125): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=1, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 2.2 in stage 14.0 (TID 53, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 125): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=2, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:======> (1 + 2) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:======> (1 + 2) / 3][Stage 10:======> (1 + 2) / 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1) / 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1) / 3]
[Stage 10:======================================> (2 + 1) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:>
(0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:===================> (1 + 2) / 3]
[Stage 11:======================================> (2 + 1) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:===================>
(1 + 2) / 3]
[Stage 12:======================================> (2 + 1) / 3]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:=======> (1 + 7) / 8]
[Stage 13:==============> (2 + 6) / 8]
[Stage 13:=====================> (3 + 5)
/ 8]
[Stage 13:=============================> (4 + 4) / 8]
[Stage 13:====================================> (5 + 3) / 8]
[Stage 13:===========================================> (6 + 2) / 8]
[Stage 13:==================================================> (7 + 1) / 8]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [HeartbeatReceiver] Removing executor 125 with no recent heartbeats: 177831 ms exceeds timeout 120000 ms
[ERROR] [TaskSchedulerImpl] Lost executor 125 on piotrain1.profession.hu<http://piotrain1.profession.hu>: Executor heartbeat timed out after 177831 ms
[WARN] [TaskSetManager] Lost task 0.0 in stage 14.1 (TID 74, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 125): ExecutorLostFailure (executor 125 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 177831 ms
[WARN] [TaskSetManager] Lost task 2.0 in stage 14.1 (TID 76, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 125): ExecutorLostFailure (executor 125 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 177831 ms
[WARN] [TaskSetManager] Lost task 1.0 in stage 14.1 (TID 75, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 125): ExecutorLostFailure (executor 125 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 177831 ms
[WARN] [TransportChannelHandler] Exception in connection from /172.31.23.58:34448<http://172.31.23.58:34448>
[ERROR] [TaskSchedulerImpl] Lost executor 125 on piotrain1.profession.hu<http://piotrain1.profession.hu>: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 1.1 in stage 14.1 (TID 77, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 125): ExecutorLostFailure (executor 125 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 0.1 in stage 14.1 (TID 79, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 125): ExecutorLostFailure (executor 125 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 2.1 in stage 14.1 (TID 78, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 125): ExecutorLostFailure (executor 125 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[Stage 14:> (0 + 0) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [TaskSetManager] Lost task 1.2 in stage 14.1 (TID 82, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 257): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=1, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 0.2 in stage 14.1 (TID 81, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 257): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=0, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 2.2 in stage 14.1 (TID 80, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 257): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=2, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:> (0 + 3) / 3][Stage 10:======> (1 + 2) / 3]
[Stage 9:======> (1 + 2) / 3][Stage 10:======> (1 + 2) / 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1) / 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:>
(0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:===================> (1 + 2) / 3]
[Stage 11:======================================> (2 + 1) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:===================> (1 + 2) / 3]
[Stage 12:======================================> (2 + 1) / 3]
[Stage 13:>
(0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:=======> (1 + 7) / 8]
[Stage 13:==============> (2 + 6) / 8]
[Stage 13:=====================> (3 + 5) / 8]
[Stage 13:=============================> (4 + 4) / 8]
[Stage 13:====================================> (5 + 3)
/ 8]
[Stage 13:===========================================> (6 + 2) / 8]
[Stage 13:==================================================> (7 + 1) / 8]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [HeartbeatReceiver] Removing executor 257 with no recent heartbeats: 146964 ms exceeds timeout 120000 ms
[ERROR] [TaskSchedulerImpl] Lost executor 257 on piotrain1.profession.hu<http://piotrain1.profession.hu>: Executor heartbeat timed out after 146964 ms
[WARN] [TaskSetManager] Lost task 1.0 in stage 14.2 (TID 104, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 257): ExecutorLostFailure (executor 257 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 146964 ms
[WARN] [TaskSetManager] Lost task 0.0 in stage 14.2 (TID 103, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 257): ExecutorLostFailure (executor 257 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 146964 ms
[WARN] [TaskSetManager] Lost task 2.0 in stage 14.2 (TID 105, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 257): ExecutorLostFailure (executor 257 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 146964 ms
[Stage 14:> (0 + 3) / 3][WARN] [TransportChannelHandler] Exception in connection from /172.31.23.58:34522<http://172.31.23.58:34522>
[ERROR] [TaskSchedulerImpl] Lost executor 257 on piotrain1.profession.hu<http://piotrain1.profession.hu>: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 0.1 in stage 14.2 (TID 107, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 257): ExecutorLostFailure (executor 257 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 2.1 in stage 14.2 (TID 106, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 257): ExecutorLostFailure (executor 257 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 1.1 in stage 14.2 (TID 108, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 257): ExecutorLostFailure (executor 257 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[Stage 14:> (0 + 0) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [TaskSetManager] Lost task 1.2 in stage 14.2 (TID 109, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 379): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=1, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 0.2 in stage 14.2 (TID 111, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 379): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=0, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 2.2 in stage 14.2 (TID 110, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 379): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=2, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3) / 3]
[Stage 9:======> (1 + 2) / 3][Stage 10:======> (1 + 2) / 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:======> (1 + 2) / 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1) / 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1) / 3]
[Stage 9:=======================================> (2 + 1) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:>
(0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:> (0 + 3) / 3]
[Stage 11:===================> (1 + 2) / 3]
[Stage 11:======================================> (2 + 1) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:> (0 + 3) / 3]
[Stage 12:===================>
(1 + 2) / 3]
[Stage 12:======================================> (2 + 1) / 3]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:> (0 + 8) / 8]
[Stage 13:=======> (1 + 7) / 8]
[Stage 13:==============> (2 + 6) / 8]
[Stage 13:=============================> (4 + 4)
/ 8]
[Stage 13:====================================> (5 + 3) / 8]
[Stage 13:===========================================> (6 + 2) / 8]
[Stage 13:==================================================> (7 + 1) / 8]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [HeartbeatReceiver] Removing executor 379 with no recent heartbeats: 164583 ms exceeds timeout 120000 ms
[ERROR] [TaskSchedulerImpl] Lost executor 379 on piotrain1.profession.hu<http://piotrain1.profession.hu>: Executor heartbeat timed out after 164583 ms
[WARN] [TaskSetManager] Lost task 2.0 in stage 14.3 (TID 134, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 379): ExecutorLostFailure (executor 379 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164583 ms
[WARN] [TaskSetManager] Lost task 1.0 in stage 14.3 (TID 133, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 379): ExecutorLostFailure (executor 379 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164583 ms
[WARN] [TaskSetManager] Lost task 0.0 in stage 14.3 (TID 132, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 379): ExecutorLostFailure (executor 379 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164583 ms
[WARN] [TransportChannelHandler] Exception in connection from /172.31.23.58:34594<http://172.31.23.58:34594>
[ERROR] [TaskSchedulerImpl] Lost executor 379 on piotrain1.profession.hu<http://piotrain1.profession.hu>: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 2.1 in stage 14.3 (TID 137, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 379): ExecutorLostFailure (executor 379 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 1.1 in stage 14.3 (TID 136, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 379): ExecutorLostFailure (executor 379 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 0.1 in stage 14.3 (TID 135, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 379): ExecutorLostFailure (executor 379 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[Stage 14:> (0 + 0) / 3]
[Stage 14:> (0 + 3) / 3][WARN] [TaskSetManager] Lost task 2.2 in stage 14.3 (TID 140, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 509): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=2, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 1.2 in stage 14.3 (TID 139, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 509): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=1, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 0.2 in stage 14.3 (TID 138, piotrain1.profession.hu<http://piotrain1.profession.hu>, executor 509): FetchFailed(null, shuffleId=5, mapId=-1, reduceId=0, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[INFO] [ServerConnector] Stopped Spark@5820070c{HTTP/1.1}{0.0.0.0:4040<http://0.0.0.0:4040>}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@61ae0d43{/stages/stage/kill,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@74024f3{/jobs/job/kill,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@9b5f3c7{/api,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@1b0e9707{/,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@5dc769f9{/static,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@1c52552f{/executors/threadDump/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@671ea6ff{/executors/threadDump,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@16da1abc{/executors/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@2b4d4327{/executors,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@5f34907b{/environment/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@3d299393{/environment,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@4b1ec694{/storage/rdd/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@65e22def{/storage/rdd,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@5426cb36{/storage/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@20ead579{/storage,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@63f9b562{/stages/pool/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@6cbbb9c4{/stages/pool,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@daf22f0{/stages/stage/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@54b2fc58{/stages/stage,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@7a6ea47d{/stages/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@158e9f6e{/stages,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@7645f03e{/jobs/job/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@6c184d4d{/jobs/job,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@173f1614{/jobs/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@51a651c1{/jobs,null,UNAVAILABLE,@Spark}
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: ResultStage 14 (foreachPartition at HBPEvents.scala:117) has failed the maximum allowable number of times: 4. Most recent failure reason: org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 5
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.apache.spark.scheduler.DAGScheduler.org<http://org.apache.spark.scheduler.DAGScheduler.org>$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1262)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1925)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1938)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1951)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1965)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:926)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:924)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:924)
at org.apache.predictionio.data.storage.hbase.HBPEvents.delete(HBPEvents.scala:117)
at org.apache.predictionio.core.SelfCleaningDataSource$class.removePEvents(SelfCleaningDataSource.scala:198)
at com.actionml.DataSource.removePEvents(DataSource.scala:48)
at org.apache.predictionio.core.SelfCleaningDataSource$class.wipePEvents(SelfCleaningDataSource.scala:184)
at com.actionml.DataSource.wipePEvents(DataSource.scala:48)
at org.apache.predictionio.core.SelfCleaningDataSource$class.cleanPersistedPEvents(SelfCleaningDataSource.scala:170)
at com.actionml.DataSource.cleanPersistedPEvents(DataSource.scala:48)
at com.actionml.DataSource.readTraining(DataSource.scala:70)
at com.actionml.DataSource.readTraining(DataSource.scala:48)
at org.apache.predictionio.controller.PDataSource.readTrainingBase(PDataSource.scala:40)
at org.apache.predictionio.controller.Engine$.train(Engine.scala:642)
at org.apache.predictionio.controller.Engine.train(Engine.scala:176)
at org.apache.predictionio.workflow.CoreWorkflow$.runTrain(CoreWorkflow.scala:67)
at org.apache.predictionio.workflow.CreateWorkflow$.main(CreateWorkflow.scala:251)
at org.apache.predictionio.workflow.CreateWorkflow.main(CreateWorkflow.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
real 138m25.051s
user 0m45.616s
sys 0m5.844s
PIO -== Profession ==- train done.
Tue Jul 24 20:48:45 CEST 2018
Best regards,
Adam Krajcs
Re: UR template train doubled the events number
Posted by Pat Ferrel <pa...@occamsmachete.com>.
Don’t use the DB cleaning feature inside the template’s engine.json. It
takes too much memory and time to do for each `pio train` we have a
template that only does cleaning: https://github.com/actionml/db-cleaner with
docs here: http://actionml.com/docs/db_cleaner_template
The reason the events doubled is probably because the clean creates them
before dropping the unlceaned ones but dies due to lack of memory.
From: Krajcs Ádám <ad...@profession.hu> <ad...@profession.hu>
Reply: user@predictionio.apache.org <us...@predictionio.apache.org>
<us...@predictionio.apache.org>
Date: July 25, 2018 at 12:39:13 AM
To: user@predictionio.apache.org <us...@predictionio.apache.org>
<us...@predictionio.apache.org>
Subject: UR template train doubled the events number
Hi everybody,
We use predictionIO 0.12.1 and UR template 0.7.2. During the train the
events number doubled and the train fa
iled. I’ve never seen anything like this. What happened?
Here are the event numbers during the train:
2018-07-23 18:40:01 - pio_event:events_1 - 9885028
2018-07-23 18:45:01 - pio_event:events_1 - 9886805
2018-07-23 18:50:01 - pio_event:events_1 - 11553133
2018-07-23 18:55:01 - pio_event:events_1 - 17096412
2018-07-23 19:00:01 - pio_event:events_1 – 17618479
And the output of the train:
Tue Jul 24 18:30:20 CEST 2018
Start -== Profession ==- PIO train...
=== SPARK Gui: ===>
[INFO] [Runner$] Submission command:
/opt/PredictionIO-0.12.1/vendors/spark/bin/spark-submit --driver-memory 6G
--master spark://xxx.xxx.hu:7077 --class
org.apache.predictionio.workflow.CreateWorkflow --jars
file:/home/pio/UR/profession-job/target/scala-2.11/universal-recommender_2.11-0.7.2.jar,file:/home/pio/UR/profession-job/target/scala-2.11/universal-recommender-assembly-0.7.2-deps.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-hdfs-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-s3-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-jdbc-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-hbase-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-elasticsearch-assembly-0.12.1.jar,file:/opt/PredictionIO-0.12.1/lib/spark/pio-data-localfs-assembly-0.12.1.jar
--files
file:/opt/PredictionIO-0.12.1/conf/log4j.properties,file:/opt/PredictionIO-0.12.1/vendors/elasticsearch/config/elasticsearch.yml,file:/
opt/PredictionIO-0.12.1/vendors/hbase/conf/hbase-site.xml
--driver-class-path
/opt/PredictionIO-0.12.1/conf:/opt/PredictionIO-0.12.1/vendors/elasticsearch/config:/opt/PredictionIO-0.12.1/vendors/hbase/conf
--driver-java-options -Dpio.log.dir=/var/log/predictionio/
file:/opt/PredictionIO-0.12.1/lib/pio-assembly-0.12.1.jar --engine-id
com.actionml.RecommendationEngine --engine-version
7c5332885ce2f37d0f45f8068f57af06eb76b290 --engine-variant
file:/home/pio/UR/profession-job/engine.json --verbosity 0 --json-extractor
Both --env
PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/pio/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=
xxx.xxx.hu
,PIO_STORAGE_SOURCES_HBASE_HOME=/opt/PredictionIO-0.12.1/vendors/hbase,PIO_HOME=/opt/PredictionIO-0.12.1,PIO_FS_ENGINESDIR=/home/pio/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/home/pio/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_S
TORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=/opt/PredictionIO-0.12.1/vendors/elasticsearch,PIO_FS_TMPDIR=/home/pio/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_LOG_DIR=/var/log/predictionio/,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/opt/PredictionIO-0.12.1/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
[INFO] [RecommendationEngine$]
_ _ __ __ _
/\ | | (_) | \/ | |
/ \ ___| |_ _ ___ _ __ | \ / | |
/ /\ \ / __| __| |/ _ \| '_ \| |\/| | |
/ ____ \ (__| |_| | (_) | | | | | | | |____
/_/ \_\___|\__|_|\___/|_| |_|_| |_|______|
[INFO] [Engine] Extracting datasource params...
[INFO] [WorkflowUtils$] No 'name' is found. Default empty String will be
used.
[INFO] [Engine] Datasource params:
(,DataSourceParams(professionJob201806,List(view,
jobapply),Some(EventWindow(Some(26 days),true,false)),None))
[INFO] [Engine] Extracting preparator params...
[INFO] [Engine] Preparator params: (,Empty)
[INFO] [Engine] Extracting serving params...
[INFO] [Engine] Serving params: (,Empty)
[INFO] [log] Logging initialized @6782ms
[INFO] [Server] jetty-9.2.z-SNAPSHOT
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@51a651c1
{/jobs,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@173f1614
{/jobs/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@6c184d4d
{/jobs/job,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@7645f03e
{/jobs/job/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@158e9f6e
{/stages,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@7a6ea47d
{/stages/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@54b2fc58
{/stages/stage,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@daf22f0
{/stages/stage/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@6cbbb9c4
{/stages/pool,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@63f9b562
{/stages/pool/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@20ead579
{/storage,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5426cb36
{/storage/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@65e22def
{/storage/rdd,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@4b1ec694
{/storage/rdd/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@3d299393
{/environment,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5f34907b
{/environment/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2b4d4327
{/executors,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@16da1abc
{/executors/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@671ea6ff
{/executors/threadDump,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@1c52552f
{/executors/threadDump/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5dc769f9
{/static,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@1b0e9707
{/,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@9b5f3c7
{/api,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@74024f3
{/jobs/job/kill,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@61ae0d43
{/stages/stage/kill,null,AVAILABLE,@Spark}
[INFO] [ServerConnector] Started Spark@5820070c{HTTP/1.1}{0.0.0.0:4040}
[INFO] [Server] Started @7032ms
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@245253d8
{/metrics/json,null,AVAILABLE,@Spark}
[INFO] [DataSource]
╔════════════════════════════════════════════════════════════╗
║ Init DataSource ║
║ ══════════════════════════════════════════════════════════ ║
║ App name professionJob201806 ║
║ Event window Some(EventWindow(Some(26 days),true,false))
║
║ Event names List(view, jobapply) ║
║ Min events per user None ║
╚════════════════════════════════════════════════════════════╝
[INFO] [URAlgorithm]
╔════════════════════════════════════════════════════════════╗
║ Init URAlgorithm ║
║ ══════════════════════════════════════════════════════════ ║
║ App name professionJob201806 ║
║ ES index name urindexsecondary ║
║ ES type name items ║
║ RecsModel all ║
║ Event names List(view, jobapply) ║
║ ══════════════════════════════════════════════════════════ ║
║ Random seed -853494969 ║
║ MaxCorrelatorsPerEventType 50 ║
║ MaxEventsPerEventType 500 ║
║ BlacklistEvents List(jobapply) ║
║ ══════════════════════════════════════════════════════════ ║
║ User bias 1.0 ║
║ Item bias 1.0 ║
║ Max query events 40 ║
║ Limit 20 ║
║ ══════════════════════════════════════════════════════════ ║
║ Rankings: ║
║ popular Some(popRank) ║
╚════════════════════════════════════════════════════════════╝
[INFO] [Engine$] EngineWorkflow.train
[INFO] [Engine$] DataSource: com.actionml.DataSource@5b35c7d
[INFO] [Engine$] Preparator: com.actionml.Preparator@56826a75
[INFO] [Engine$] AlgorithmList: List(com.actionml.URAlgorithm@14d25b6e)
[INFO] [Engine$] Data sanity check is on.
[Stage 0:> (0 + 3)
/ 3]
[Stage 0:> (0 + 3)
/ 3]
[Stage 0:===================> (1 + 2)
/ 3]
[Stage 0:=======================================> (2 + 1)
/ 3]
[Stage 0:=======================================> (2 + 1)
/ 3]
[Stage 1:> (0 + 3)
/ 3]
[Stage 1:> (0 + 3)
/ 3]
[Stage 1:===================> (1 + 2)
/ 3]
[Stage 1:=======================================> (2 + 1)
/ 3]
[Stage 1:=======================================> (2 + 1)
/ 3]
[Stage 2:> (0 + 3)
/ 3]
[Stage 2:>
(0 + 3) / 3]
[Stage 2:> (0 + 3)
/ 3]
[Stage 2:> (0 + 3)
/ 3]
[Stage 2:> (0 + 3)
/ 3]
[Stage 2:> (0 + 3)
/ 3]
[Stage 2:===================> (1 + 2)
/ 3]
[Stage 2:=======================================> (2 + 1)
/ 3]
[Stage 3:> (0 + 3) / 3][Stage 5:> (0 + 3)
/ 3]
[Stage 3:> (0 + 3) / 3][Stage 5:> (0 + 3)
/ 3]
[Stage 3:> (0 + 3) / 3][Stage 5:> (0 + 3)
/ 3]
[Stage 3:> (0 + 3) / 3][Stage 5:> (0 + 3)
/ 3]
[Stage 3:> (0 + 3) / 3][Stage 5
:> (0 + 3) / 3]
[Stage 3:> (0 + 3) / 3][Stage 5:======> (1 + 2)
/ 3]
[Stage 3:======> (1 + 2) / 3][Stage 5:======> (1 + 2)
/ 3]
[Stage 3:============> (2 + 1) / 3][Stage 5:======> (1 + 2)
/ 3]
[Stage 3:============> (2 + 1) / 3][Stage 5:============> (2 + 1)
/ 3]
[Stage 3:============> (2 + 1) / 3][Stage 6:> (0 + 0)
/ 3]
[Stage 3:=======================================> (2 + 1)
/ 3]
[Stage 3:============> (2 + 1) / 3][Stage 6:> (0 + 3)
/ 3]
[Stage 3:============> (2 + 1) / 3][Stage 6:> (0 + 3)
/ 3]
[Stage 3:============> (2 + 1) / 3][Stage 6:> (0 + 3)
/ 3]
[Stage 6:> (0 + 3)
/ 3]
[Stage 6:> (0 + 3)
/ 3]
[Stage 6:===================> (1 +
2) / 3]
[Stage 6:=======================================> (2 + 1)
/ 3]
[Stage 7:> (0 + 8)
/ 8]
[Stage 7:> (0 + 8)
/ 8]
[Stage 7:> (0 + 8)
/ 8]
[Stage 7:> (0 + 8)
/ 8]
[Stage 7:> (0 + 8)
/ 8]
[Stage 7:> (0 + 8)
/ 8]
[Stage 7:> (0 + 8)
/ 8]
[Stage 7:=======> (1 + 7)
/ 8]
[Stage 7:==============> (2 + 6)
/ 8]
[Stage 7:======================> (3 + 5)
/ 8]
[Stage 7:=============================> (4 + 4)
/ 8]
[Stage 7:========
============================> (5 + 3) / 8]
[Stage 7:============================================> (6 + 2)
/ 8]
[Stage 8:> (0 + 8)
/ 8]
[Stage 8:> (0 + 8)
/ 8]
[Stage 8:> (0 + 8)
/ 8]
[Stage 8:> (0 + 8)
/ 8]
[Stage 8:=======> (1 + 7)
/ 8]
[Stage 8:==============> (2 + 6)
/ 8]
[Stage 8:======================> (3 + 5)
/ 8]
[Stage 8:=============================> (4 + 4)
/ 8]
[Stage 8:====================================> (5 + 3)
/ 8]
[Stage 8:===================================================> (7 + 1)
/ 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5)
/ 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5)
/ 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5)
/ 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5)
/ 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5)
/ 8]
[Stage 9:> (0 + 3) / 3][Stage 13:> (0 + 5)
/ 8]
[Stage 9:> (0 + 3) / 3][Stage 13:==> (1 + 5)
/ 8]
[Stage 9:> (0 + 3) / 3][Stage 13:======> (3 + 5)
/ 8]
[Stage 9:> (0 + 3) / 3][Stage 13:=========> (4 + 4)
/ 8]
[Stage 9:> (0 + 3) / 3][Stage 13:===========> (5 + 3)
/ 8]
[Stage 9:======> (1 + 2) / 3][Stage 13:===========> (5 + 3)
/ 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:===========>
(5 + 3) / 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:===========> (5 + 3)
/ 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:===========> (5 + 3)
/ 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:=============> (6 + 2)
/ 8]
[Stage 9:============> (2 + 1) / 3][Stage 13:===============> (7 + 1)
/ 8]
[Stage 9:=======================================> (2 + 1)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:
> (0 + 3) / 3]
[Stage 14:> (0 + 3)
/ 3][WARN] [HeartbeatReceiver] Removing executor 0 with no recent
heartbeats: 172452 ms exceeds timeout 120000 ms
[ERROR] [TaskSchedulerImpl] Lost executor 0 on piotrain1.profession.hu:
Executor heartbeat timed out after 172452 ms
[WARN] [TaskSetManager] Lost task 2.0 in stage 14.0 (TID 47,
piotrain1.profession.hu, executor 0): ExecutorLostFailure (executor 0
exited caused by one of the running tasks) Reason: Executor heartbeat timed
out after 172452 ms
[WARN] [TaskSetManager] Lost task 1.0 in stage 14.0 (TID 46,
piotrain1.profession.hu, executor 0): ExecutorLostFailure (executor 0
exited caused by one of the running tasks) Reason: Executor heartbeat timed
out after 172452 ms
[WARN] [TaskSetManager] Lost task 0.0 in stage 14.0 (TID 45,
piotrain1.profession.hu, executor 0): ExecutorLostFailure (executor 0
exited caused by one of the running tasks) Reason: Executor heartbeat timed
out after 172452 ms
[WARN] [TransportChannelHandler] Exception in connection from /
172.31.23.58:34288
[ERROR] [TaskSchedulerImpl] Lost executor 0 on piotrain1.profession.hu:
Remote RPC client disassociated. Likely due to containers exceeding
thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 2.1 in stage 14.0 (TID 50,
piotrain1.profession.hu, executor 0): ExecutorLostFailure (executor 0
exited caused by one of the running tasks) Reason: Remote RPC client
disassociated. Likely due to containers exceeding thresholds, or network
issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 1.1 in stage 14.0 (TID 49,
piotrain1.profession.hu, executor 0): ExecutorLostFailure (executor 0
exited caused by one of the running tasks) Reason: Remote RPC client
disassociated. Likely due to containers exceeding thresholds, or network
issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 0.1 in stage 14.0 (TID 48,
piotrain1.profession.hu, executor 0): ExecutorLostFailure (executor 0
exited caused by one of the running tasks) Reason: Remote RPC client
disassociated. Likely due to containers exceeding thresholds, or network
issues. Check driver logs for WARN messages.
[Stage 14:> (0 + 0)
/ 3]
[Stage 14:> (0 + 3)
/ 3][WARN] [TaskSetManager] Lost task 0.2 in stage 14.0 (TID 51,
piotrain1.profession.hu, executor 125): FetchFailed(null, shuffleId=5,
mapId=-1, reduceId=0, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 5
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at
org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at
org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at
org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at
org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 1.2 in stage 14.0 (TID 52,
piotrain1.profession.hu, executor 125): FetchFailed(null, shuffleId=5,
mapId=-1, reduceId=1, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 5
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at
org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at
org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at
org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at
org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 2.2 in stage 14.0 (TID 53,
piotrain1.profession.hu, executor 125): FetchFailed(null, shuffleId=5,
mapId=-1, reduceId=2, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 5
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at
org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at
org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at
org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at
org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3)
/ 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3)
/ 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3)
/ 3]
[Stage 9:======> (1 + 2) / 3][Stage 10:> (0 + 3)
/ 3]
[Stage 9:======> (1 + 2) / 3][Stage 10:======> (1 + 2)
/ 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1)
/ 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1)
/ 3]
[Stage 10:======================================> (2 + 1)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:>
(0 + 3) / 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:===================> (1 + 2)
/ 3]
[Stage 11:======================================> (2 + 1)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:===================>
(1 + 2) / 3]
[Stage 12:======================================> (2 + 1)
/ 3]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:=======> (1 + 7)
/ 8]
[Stage 13:==============> (2 + 6)
/ 8]
[Stage 13:=====================> (3 + 5)
/ 8]
[Stage 13:=============================> (4 + 4)
/ 8]
[Stage 13:====================================> (5 + 3)
/ 8]
[Stage 13:===========================================> (6 + 2)
/ 8]
[Stage 13:==================================================> (7 + 1)
/ 8]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3][WARN] [HeartbeatReceiver] Removing executor 125 with no recent
heartbeats: 177831 ms exceeds timeout 120000 ms
[ERROR] [TaskSchedulerImpl] Lost executor 125 on piotrain1.profession.hu:
Executor heartbeat timed out after 177831 ms
[WARN] [TaskSetManager] Lost task 0.0 in stage 14.1 (TID 74,
piotrain1.profession.hu, executor 125): ExecutorLostFailure (executor 125
exited caused by one of the running tasks) Reason: Executor heartbeat timed
out after 177831 ms
[WARN] [TaskSetManager] Lost task 2.0 in stage 14.1 (TID 76,
piotrain1.profession.hu, executor 125): ExecutorLostFailure (executor 125
exited caused by one of the running tasks) Reason: Executor heartbeat timed
out after 177831 ms
[WARN] [TaskSetManager] Lost task 1.0 in stage 14.1 (TID 75,
piotrain1.profession.hu, executor 125): ExecutorLostFailure (executor 125
exited caused by one of the running tasks) Reason: Executor heartbeat timed
out after 177831 ms
[WARN] [TransportChannelHandler] Exception in connection from /
172.31.23.58:34448
[ERROR] [TaskSchedulerImpl] Lost executor 125 on piotrain1.profession.hu:
Remote RPC client disassociated. Likely due to containers exceeding
thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 1.1 in stage 14.1 (TID 77,
piotrain1.profession.hu, executor 125): ExecutorLostFailure (executor 125
exited caused by one of the running tasks) Reason: Remote RPC client
disassociated. Likely due to containers exceeding thresholds, or network
issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 0.1 in stage 14.1 (TID 79,
piotrain1.profession.hu, executor 125): ExecutorLostFailure (executor 125
exited caused by one of the running tasks) Reason: Remote RPC client
disassociated. Likely due to containers exceeding thresholds, or network
issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 2.1 in stage 14.1 (TID 78,
piotrain1.profession.hu, executor 125): ExecutorLostFailure (executor 125
exited caused by one of the running tasks) Reason: Remote RPC client
disassociated. Likely due to containers exceeding thresholds, or network
issues. Check driver logs for WARN messages.
[Stage 14:> (0 + 0)
/ 3]
[Stage 14:> (0 + 3)
/ 3][WARN] [TaskSetManager] Lost task 1.2 in stage 14.1 (TID 82,
piotrain1.profession.hu, executor 257): FetchFailed(null, shuffleId=5,
mapId=-1, reduceId=1, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 5
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at
org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at
org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at
org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at
org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 0.2 in stage 14.1 (TID 81,
piotrain1.profession.hu, executor 257): FetchFailed(null, shuffleId=5,
mapId=-1, reduceId=0, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 5
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at
org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at
org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at
org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at
org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 2.2 in stage 14.1 (TID 80,
piotrain1.profession.hu, executor 257): FetchFailed(null, shuffleId=5,
mapId=-1, reduceId=2, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 5
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at
org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at
org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at
org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at
org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3)
/ 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3)
/ 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3)
/ 3]
[Stage 9:> (0 + 3) / 3][Stage 10:======> (1 + 2)
/ 3]
[Stage 9:======> (1 + 2) / 3][Stage 10:======> (1 + 2)
/ 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1)
/ 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:>
(0 + 3) / 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:===================> (1 + 2)
/ 3]
[Stage 11:======================================> (2 + 1)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:===================> (1 + 2)
/ 3]
[Stage 12:======================================> (2 + 1)
/ 3]
[Stage 13:>
(0 + 8) / 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:=======> (1 + 7)
/ 8]
[Stage 13:==============> (2 + 6)
/ 8]
[Stage 13:=====================> (3 + 5)
/ 8]
[Stage 13:=============================> (4 + 4)
/ 8]
[Stage 13:====================================> (5 + 3)
/ 8]
[Stage 13:===========================================> (6 + 2)
/ 8]
[Stage 13:==================================================> (7 + 1)
/ 8]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3][WARN] [HeartbeatReceiver] Removing executor 257 with no recent
heartbeats: 146964 ms exceeds timeout 120000 ms
[ERROR] [TaskSchedulerImpl] Lost executor 257 on piotrain1.profession.hu:
Executor heartbeat timed out after 146964 ms
[WARN] [TaskSetManager] Lost task 1.0 in stage 14.2 (TID 104,
piotrain1.profession.hu, executor 257): ExecutorLostFailure (executor 257
exited caused by one of the running tasks) Reason: Executor heartbeat timed
out after 146964 ms
[WARN] [TaskSetManager] Lost task 0.0 in stage 14.2 (TID 103,
piotrain1.profession.hu, executor 257): ExecutorLostFailure (executor 257
exited caused by one of the running tasks) Reason: Executor heartbeat timed
out after 146964 ms
[WARN] [TaskSetManager] Lost task 2.0 in stage 14.2 (TID 105,
piotrain1.profession.hu, executor 257): ExecutorLostFailure (executor 257
exited caused by one of the running tasks) Reason: Executor heartbeat timed
out after 146964 ms
[Stage 14:> (0 + 3)
/ 3][WARN] [TransportChannelHandler] Exception in connection from /
172.31.23.58:34522
[ERROR] [TaskSchedulerImpl] Lost executor 257 on piotrain1.profession.hu:
Remote RPC client disassociated. Likely due to containers exceeding
thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 0.1 in stage 14.2 (TID 107,
piotrain1.profession.hu, executor 257): ExecutorLostFailure (executor 257
exited caused by one of the running tasks) Reason: Remote RPC client
disassociated. Likely due to containers exceeding thresholds, or network
issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 2.1 in stage 14.2 (TID 106,
piotrain1.profession.hu, executor 257): ExecutorLostFailure (executor 257
exited caused by one of the running tasks) Reason: Remote RPC client
disassociated. Likely due to containers exceeding thresholds, or network
issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 1.1 in stage 14.2 (TID 108,
piotrain1.profession.hu, executor 257): ExecutorLostFailure (executor 257
exited caused by one of the running tasks) Reason: Remote RPC client
disassociated. Likely due to containers exceeding thresholds, or network
issues. Check driver logs for WARN messages.
[Stage 14:> (0 + 0)
/ 3]
[Stage 14:> (0 + 3)
/ 3][WARN] [TaskSetManager] Lost task 1.2 in stage 14.2 (TID 109,
piotrain1.profession.hu, executor 379): FetchFailed(null, shuffleId=5,
mapId=-1, reduceId=1, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 5
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at
org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at
org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at
org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at
org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 0.2 in stage 14.2 (TID 111,
piotrain1.profession.hu, executor 379): FetchFailed(null, shuffleId=5,
mapId=-1, reduceId=0, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 5
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at
org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at
org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at
org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at
org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 2.2 in stage 14.2 (TID 110,
piotrain1.profession.hu, executor 379): FetchFailed(null, shuffleId=5,
mapId=-1, reduceId=2, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 5
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at
org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at
org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at
org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at
org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3)
/ 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3)
/ 3]
[Stage 9:> (0 + 3) / 3][Stage 10:> (0 + 3)
/ 3]
[Stage 9:======> (1 + 2) / 3][Stage 10:======> (1 + 2)
/ 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:======> (1 + 2)
/ 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1)
/ 3]
[Stage 9:============> (2 + 1) / 3][Stage 10:============> (2 + 1)
/ 3]
[Stage 9:=======================================> (2 + 1)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:>
(0 + 3) / 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:> (0 + 3)
/ 3]
[Stage 11:===================> (1 + 2)
/ 3]
[Stage 11:======================================> (2 + 1)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:> (0 + 3)
/ 3]
[Stage 12:===================>
(1 + 2) / 3]
[Stage 12:======================================> (2 + 1)
/ 3]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:> (0 + 8)
/ 8]
[Stage 13:=======> (1 + 7)
/ 8]
[Stage 13:==============> (2 + 6)
/ 8]
[Stage 13:=============================> (4 + 4)
/ 8]
[Stage 13:====================================> (5 + 3)
/ 8]
[Stage 13:===========================================> (6 + 2)
/ 8]
[Stage 13:==================================================> (7 + 1)
/ 8]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3]
[Stage 14:> (0 + 3)
/ 3][WARN] [HeartbeatReceiver] Removing executor 379 with no recent
heartbeats: 164583 ms exceeds timeout 120000 ms
[ERROR] [TaskSchedulerImpl] Lost executor 379 on piotrain1.profession.hu:
Executor heartbeat timed out after 164583 ms
[WARN] [TaskSetManager] Lost task 2.0 in stage 14.3 (TID 134,
piotrain1.profession.hu, executor 379): ExecutorLostFailure (executor 379
exited caused by one of the running tasks) Reason: Executor heartbeat timed
out after 164583 ms
[WARN] [TaskSetManager] Lost task 1.0 in stage 14.3 (TID 133,
piotrain1.profession.hu, executor 379): ExecutorLostFailure (executor 379
exited caused by one of the running tasks) Reason: Executor heartbeat timed
out after 164583 ms
[WARN] [TaskSetManager] Lost task 0.0 in stage 14.3 (TID 132,
piotrain1.profession.hu, executor 379): ExecutorLostFailure (executor 379
exited caused by one of the running tasks) Reason: Executor heartbeat timed
out after 164583 ms
[WARN] [TransportChannelHandler] Exception in connection from /
172.31.23.58:34594
[ERROR] [TaskSchedulerImpl] Lost executor 379 on piotrain1.profession.hu:
Remote RPC client disassociated. Likely due to containers exceeding
thresholds, or network issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 2.1 in stage 14.3 (TID 137,
piotrain1.profession.hu, executor 379): ExecutorLostFailure (executor 379
exited caused by one of the running tasks) Reason: Remote RPC client
disassociated. Likely due to containers exceeding thresholds, or network
issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 1.1 in stage 14.3 (TID 136,
piotrain1.profession.hu, executor 379): ExecutorLostFailure (executor 379
exited caused by one of the running tasks) Reason: Remote RPC client
disassociated. Likely due to containers exceeding thresholds, or network
issues. Check driver logs for WARN messages.
[WARN] [TaskSetManager] Lost task 0.1 in stage 14.3 (TID 135,
piotrain1.profession.hu, executor 379): ExecutorLostFailure (executor 379
exited caused by one of the running tasks) Reason: Remote RPC client
disassociated. Likely due to containers exceeding thresholds, or network
issues. Check driver logs for WARN messages.
[Stage 14:> (0 + 0)
/ 3]
[Stage 14:> (0 + 3)
/ 3][WARN] [TaskSetManager] Lost task 2.2 in stage 14.3 (TID 140,
piotrain1.profession.hu, executor 509): FetchFailed(null, shuffleId=5,
mapId=-1, reduceId=2, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 5
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at
org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at
org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at
org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at
org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 1.2 in stage 14.3 (TID 139,
piotrain1.profession.hu, executor 509): FetchFailed(null, shuffleId=5,
mapId=-1, reduceId=1, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 5
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at
org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at
org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at
org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at
org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[WARN] [TaskSetManager] Lost task 0.2 in stage 14.3 (TID 138,
piotrain1.profession.hu, executor 509): FetchFailed(null, shuffleId=5,
mapId=-1, reduceId=0, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 5
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at
org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at
org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at
org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at
org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
)
[INFO] [ServerConnector] Stopped Spark@5820070c{HTTP/1.1}{0.0.0.0:4040}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@61ae0d43
{/stages/stage/kill,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@74024f3
{/jobs/job/kill,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@9b5f3c7
{/api,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@1b0e9707
{/,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@5dc769f9
{/static,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@1c52552f
{/executors/threadDump/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@671ea6ff
{/executors/threadDump,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@16da1abc
{/executors/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@2b4d4327
{/executors,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@5f34907b
{/environment/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@3d299393
{/environment,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@4b1ec694
{/storage/rdd/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@65e22def
{/storage/rdd,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@5426cb36
{/storage/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@20ead579
{/storage,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@63f9b562
{/stages/pool/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@6cbbb9c4
{/stages/pool,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@daf22f0
{/stages/stage/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@54b2fc58
{/stages/stage,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@7a6ea47d
{/stages/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@158e9f6e
{/stages,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@7645f03e
{/jobs/job/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@6c184d4d
{/jobs/job,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@173f1614
{/jobs/json,null,UNAVAILABLE,@Spark}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@51a651c1
{/jobs,null,UNAVAILABLE,@Spark}
Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: ResultStage 14 (foreachPartition at HBPEvents.scala:117)
has failed the maximum allowable number of times: 4. Most recent failure
reason: org.apache.spark.shuffle.MetadataFetchFailedException: Missing an
output location for shuffle 5
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:697)
at
org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:693)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at
org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:693)
at
org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:147)
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:49)
at
org.apache.spark.rdd.SubtractedRDD.integrate$1(SubtractedRDD.scala:113)
at
org.apache.spark.rdd.SubtractedRDD.compute(SubtractedRDD.scala:119)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1262)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
at
org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
at
org.apache.spark.SparkContext.runJob(SparkContext.scala:1925)
at
org.apache.spark.SparkContext.runJob(SparkContext.scala:1938)
at
org.apache.spark.SparkContext.runJob(SparkContext.scala:1951)
at
org.apache.spark.SparkContext.runJob(SparkContext.scala:1965)
at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:926)
at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:924)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:924)
at
org.apache.predictionio.data.storage.hbase.HBPEvents.delete(HBPEvents.scala:117)
at
org.apache.predictionio.core.SelfCleaningDataSource$class.removePEvents(SelfCleaningDataSource.scala:198)
at
com.actionml.DataSource.removePEvents(DataSource.scala:48)
at
org.apache.predictionio.core.SelfCleaningDataSource$class.wipePEvents(SelfCleaningDataSource.scala:184)
at com.actionml.DataSource.wipePEvents(DataSource.scala:48)
at
org.apache.predictionio.core.SelfCleaningDataSource$class.cleanPersistedPEvents(SelfCleaningDataSource.scala:170)
at
com.actionml.DataSource.cleanPersistedPEvents(DataSource.scala:48)
at com.actionml.DataSource.readTraining(DataSource.scala:70)
at com.actionml.DataSource.readTraining(DataSource.scala:48)
at
org.apache.predictionio.controller.PDataSource.readTrainingBase(PDataSource.scala:40)
at
org.apache.predictionio.controller.Engine$.train(Engine.scala:642)
at
org.apache.predictionio.controller.Engine.train(Engine.scala:176)
at
org.apache.predictionio.workflow.CoreWorkflow$.runTrain(CoreWorkflow.scala:67)
at
org.apache.predictionio.workflow.CreateWorkflow$.main(CreateWorkflow.scala:251)
at
org.apache.predictionio.workflow.CreateWorkflow.main(CreateWorkflow.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
real 138m25.051s
user 0m45.616s
sys 0m5.844s
PIO -== Profession ==- train done.
Tue Jul 24 20:48:45 CEST 2018
Best regards,
Adam Krajcs