You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "Raja10D (via GitHub)" <gi...@apache.org> on 2025/01/02 13:54:06 UTC
[I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Raja10D opened a new issue, #4752:
URL: https://github.com/apache/hop/issues/4752
### Apache Hop version?
2.61.0
### Java version?
17
### Operating system
Linux
### What happened?
I am reaching out to seek your assistance with an issue I am encountering while running a simple Hop pipeline using the Beam Spark pipeline engine. The pipeline executes without any issues when using the default local runner, but it fails when I switch to the Beam Spark engine.
Below are the details of my setup:
Apache Beam Version: 2.61.0
Apache Spark Version: 3.5.1
Apache Hop Version: apache-hop-client-2.11.0
The error logs are as follows:
**Error:**
2025/01/02 15:30:53 - Hop - Pipeline opened.
2025/01/02 15:30:53 - Hop - Launching pipeline [hop_trans2]...
2025/01/02 15:30:53 - Hop - Started the pipeline execution.
java.lang.NoClassDefFoundError: org/apache/spark/metrics/source/Source
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
...
2025/01/02 15:30:53 - Hop - ERROR: hop_trans2: preparing pipeline execution failed
2025/01/02 15:30:53 - Hop - ERROR: org.apache.hop.core.exception.HopException:
2025/01/02 15:30:53 - Hop - Error preparing remote pipeline
2025/01/02 15:30:53 - Hop - Error converting Hop pipeline to Beam
2025/01/02 15:30:53 - Hop -
2025/01/02 15:30:53 - Hop - Caused by: java.lang.NoClassDefFoundError: org/apache/spark/metrics/source/Source
2025/01/02 15:30:53 - Hop - at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
...
It appears that the issue is related to the org.apache.spark.metrics.source.Source class not being found.
### Issue Priority
Priority: 3
### Issue Component
Component: Hop Config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "Raja10D (via GitHub)" <gi...@apache.org>.
Raja10D commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2572391323
I have created a simple pipeline -> hop_trans2.hpl. After that I have generated a fatjar using generate fatjar in hopgui and then I have exported a metadata.json file using export metadata.json in hopgui.
After that I have used Spark submit to run my pipeline (In local runner it is running fine)
My spark-submit command
spark-submit \
--master local[4] \
--class org.apache.hop.beam.run.MainBeam \
--driver-java-options '-DPROJECT_HOME=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default' \
/home/decoders/Downloads/apache-hop-client-2.11.0/fatjar.jar \
--pipeline=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/pipelines/hop_trans2.hpl \
--metadata=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/metadata.json \
Spark
My versions:
My Hop version: 2.11.0
Apache Beam Version: 2.61.0
Apache Spark Version: 3.5.4
java 17
Error :
25/01/06 11:52:34 WARN Utils: Your hostname, decoders-Latitude-7480 resolves to a loopback address: 127.0.1.1; using 192.125.1.140 instead (on interface enp0s31f6)
25/01/06 11:52:34 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
>>>>>> Initializing Hop
Hop configuration file not found, not serializing: /home/decoders/config/hop-config.json
Error running Beam pipeline:
Error reading metadata from JSON
Error find metadata class for key null
The metadata plugin for key null could not be found in the plugin registry
org.apache.hop.core.exception.HopException:
Error reading metadata from JSON
Error find metadata class for key null
The metadata plugin for key null could not be found in the plugin registry
at org.apache.hop.core.metadata.SerializableMetadataProvider.<init>(SerializableMetadataProvider.java:143)
at org.apache.hop.beam.run.MainBeam.main(MainBeam.java:101)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1034)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hop.core.exception.HopException:
Error find metadata class for key null
The metadata plugin for key null could not be found in the plugin registry
at org.apache.hop.metadata.serializer.BaseMetadataProvider.getMetadataClassForKey(BaseMetadataProvider.java:69)
at org.apache.hop.core.metadata.SerializableMetadataProvider.<init>(SerializableMetadataProvider.java:122)
... 13 more
Caused by: org.apache.hop.core.exception.HopException:
The metadata plugin for key null could not be found in the plugin registry
at org.apache.hop.metadata.serializer.BaseMetadataProvider.getMetadataClassForKey(BaseMetadataProvider.java:61)
... 14 more
25/01/06 11:52:47 INFO ShutdownHookManager: Shutdown hook called
25/01/06 11:52:47 INFO ShutdownHookManager: Deleting directory /tmp/spark-05fb3578-b82c-4c9a-9fd2-f549997cd73e
If You know any ways to rectify this error means say it otherwise if you can run hop pipelines with Beam Spark Runner means tell me the versions you are using, I will try in that version.
Our team is working on this for 10 days, we couldn't get it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2569139330
seems to be working here...
```
Downloads/spark-3.5.4-bin-hadoop3/bin on ☁️ (eu-west-1) on ☁️ hans.van.akelyen@know.bi took 16s
❯ ./spark-submit \
--master spark://MacBook-Hans.local:7077 \
--class org.apache.hop.beam.run.MainBeam \
--driver-java-options '-DPROJECT_HOME=/Users/hans/git/hop/assemblies/client/target/hop/config/projects/samples' \
/Users/hans/git/hop/assemblies/client/target/hop/fat-jar.jar \
/Users/hans/git/hop/assemblies/client/target/hop/config/projects/samples/beam/pipelines/input-process-output.hpl \
/Users/hans/git/hop/assemblies/client/target/hop/metadata.json \
Spark
25/01/03 13:16:56 WARN Utils: Your hostname, MacBook-Hans.local resolves to a loopback address: 127.0.0.1; using 192.168.86.198 instead (on interface en0)
25/01/03 13:16:56 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
>>>>>> Initializing Hop
Hop configuration file not found, not serializing: /Users/hans/Downloads/spark-3.5.4-bin-hadoop3/bin/config/hop-config.json
Argument 1 : Pipeline filename (.hpl) : /Users/hans/git/hop/assemblies/client/target/hop/config/projects/samples/beam/pipelines/input-process-output.hpl
Argument 2 : Environment state filename: (.json) : /Users/hans/git/hop/assemblies/client/target/hop/metadata.json
Argument 3 : Pipeline run configuration : Spark
>>>>>> Loading pipeline metadata
>>>>>> Building Apache Beam Pipeline...
>>>>>> Pipeline executing starting...
2025/01/03 13:17:01 - General - Created Apache Beam pipeline with name 'input-process-output'
2025/01/03 13:17:01 - General - Handled transform (INPUT) : Customers
2025/01/03 13:17:01 - General - Handled generic transform (TRANSFORM) : Only CA, gets data from 1 previous transform(s), targets=0, infos=0
2025/01/03 13:17:01 - General - Handled generic transform (TRANSFORM) : Limit fields, re-order, gets data from 1 previous transform(s), targets=0, infos=0
2025/01/03 13:17:01 - General - Handled transform (OUTPUT) : input-process-output, gets data from Limit fields, re-order
2025/01/03 13:17:01 - General - Executing this pipeline using the Beam Pipeline Engine with run configuration 'Spark'
25/01/03 13:17:01 INFO SparkRunner: Executing pipeline using the SparkRunner.
25/01/03 13:17:01 INFO SparkContextFactory: Creating a brand new Spark Context.
25/01/03 13:17:01 INFO SparkContext: Running Spark version 3.5.4
25/01/03 13:17:01 INFO SparkContext: OS info Mac OS X, 15.2, aarch64
25/01/03 13:17:01 INFO SparkContext: Java version 17.0.11
25/01/03 13:17:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
25/01/03 13:17:02 INFO ResourceUtils: ==============================================================
25/01/03 13:17:02 INFO ResourceUtils: No custom resources configured for spark.driver.
25/01/03 13:17:02 INFO ResourceUtils: ==============================================================
25/01/03 13:17:02 INFO SparkContext: Submitted application: BeamSparkPipelineRunConfiguration
25/01/03 13:17:02 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
25/01/03 13:17:02 INFO ResourceProfile: Limiting resource is cpu
25/01/03 13:17:02 INFO ResourceProfileManager: Added ResourceProfile id: 0
25/01/03 13:17:02 INFO SecurityManager: Changing view acls to: hans
25/01/03 13:17:02 INFO SecurityManager: Changing modify acls to: hans
25/01/03 13:17:02 INFO SecurityManager: Changing view acls groups to:
25/01/03 13:17:02 INFO SecurityManager: Changing modify acls groups to:
25/01/03 13:17:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: hans; groups with view permissions: EMPTY; users with modify permissions: hans; groups with modify permissions: EMPTY
25/01/03 13:17:02 INFO Utils: Successfully started service 'sparkDriver' on port 55057.
25/01/03 13:17:02 INFO SparkEnv: Registering MapOutputTracker
25/01/03 13:17:02 INFO SparkEnv: Registering BlockManagerMaster
25/01/03 13:17:02 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
25/01/03 13:17:02 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
25/01/03 13:17:02 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
25/01/03 13:17:02 INFO DiskBlockManager: Created local directory at /private/var/folders/v0/hjm6kpgd0076ncdkpf8rg3dw0000gn/T/blockmgr-58d26b5c-d34a-4682-8fe7-b5beb2ab0873
25/01/03 13:17:02 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
25/01/03 13:17:02 INFO SparkEnv: Registering OutputCommitCoordinator
25/01/03 13:17:02 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI
25/01/03 13:17:02 INFO Utils: Successfully started service 'SparkUI' on port 4040.
25/01/03 13:17:02 INFO SparkContext: Added JAR /Users/hans/git/hop/assemblies/client/target/hop/fat-jar.jar at spark://192.168.86.198:55057/jars/fat-jar.jar with timestamp 1735906621944
25/01/03 13:17:02 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://MacBook-Hans.local:7077...
25/01/03 13:17:02 INFO TransportClientFactory: Successfully created connection to MacBook-Hans.local/127.0.0.1:7077 after 14 ms (0 ms spent in bootstraps)
25/01/03 13:17:02 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20250103131702-0002
25/01/03 13:17:02 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20250103131702-0002/0 on worker-20250103125214-192.168.86.198-54182 (192.168.86.198:54182) with 12 core(s)
25/01/03 13:17:02 INFO StandaloneSchedulerBackend: Granted executor ID app-20250103131702-0002/0 on hostPort 192.168.86.198:54182 with 12 core(s), 1024.0 MiB RAM
25/01/03 13:17:02 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 55059.
25/01/03 13:17:02 INFO NettyBlockTransferService: Server created on 192.168.86.198:55059
25/01/03 13:17:02 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
25/01/03 13:17:02 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.86.198, 55059, None)
25/01/03 13:17:02 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.86.198:55059 with 434.4 MiB RAM, BlockManagerId(driver, 192.168.86.198, 55059, None)
25/01/03 13:17:02 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.86.198, 55059, None)
25/01/03 13:17:02 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.86.198, 55059, None)
25/01/03 13:17:02 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20250103131702-0002/0 is now RUNNING
25/01/03 13:17:02 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Entering directly-translatable composite transform: 'BeamOutputTransform/TextIO.Write/WriteFiles/GatherTempFileResults/Consolidate/Reshuffle'
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Entering directly-translatable composite transform: 'BeamOutputTransform/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle'
25/01/03 13:17:02 INFO MetricsAccumulator: Instantiated metrics accumulator: MetricQueryResults()
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating Read(CompressedSource)
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.hop.beam.core.fn.StringToHopFn@7aa06fac
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.hop.beam.core.transform.TransformFn@2ea82cef
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.hop.beam.core.transform.TransformFn@5cea34e6
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.hop.beam.core.fn.HopToStringFn@526fbb80
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating Window.Assign
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesFn@2162e4a
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating GroupByKey
25/01/03 13:17:02 INFO FileBasedSource: Splitting filepattern /Users/hans/git/hop/assemblies/client/target/hop/config/projects/samples/beam/input/customers-noheader-1k.txt into bundles of size 50000 took 3 ms and produced 1 files and 2 bundles
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn@3199c2c1
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.io.WriteFiles$WriteUnshardedBundlesToTempFiles$1@7ae736a8
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating Flatten.PCollections
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.Reshuffle$AssignShardFn@4623c0d3
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Entering directly-translatable composite transform: 'BeamOutputTransform/TextIO.Write/WriteFiles/GatherTempFileResults/Consolidate/Reshuffle'
25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating Reshuffle
25/01/03 13:17:03 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.MapElements$2@2182ebc7
25/01/03 13:17:03 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.MapElements$2@447cdbaa
25/01/03 13:17:03 INFO SparkRunner$Evaluator: Evaluating View.CreatePCollectionView
25/01/03 13:17:03 INFO SparkContext: Starting job: collect at BoundedDataset.java:96
25/01/03 13:17:03 INFO DAGScheduler: Registering RDD 20 (mapToPair at GroupNonMergingWindowsFunctions.java:273) as input to shuffle 1
25/01/03 13:17:03 INFO DAGScheduler: Registering RDD 37 (repartition at GroupCombineFunctions.java:191) as input to shuffle 0
25/01/03 13:17:03 INFO DAGScheduler: Got job 0 (collect at BoundedDataset.java:96) with 4 output partitions
25/01/03 13:17:03 INFO DAGScheduler: Final stage: ResultStage 2 (collect at BoundedDataset.java:96)
25/01/03 13:17:03 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1)
25/01/03 13:17:03 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 1)
25/01/03 13:17:03 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[20] at mapToPair at GroupNonMergingWindowsFunctions.java:273), which has no missing parents
25/01/03 13:17:03 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 215.7 KiB, free 434.2 MiB)
25/01/03 13:17:03 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 69.0 KiB, free 434.1 MiB)
25/01/03 13:17:03 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.86.198:55059 (size: 69.0 KiB, free: 434.3 MiB)
25/01/03 13:17:03 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1585
25/01/03 13:17:03 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[20] at mapToPair at GroupNonMergingWindowsFunctions.java:273) (first 15 tasks are for partitions Vector(0, 1))
25/01/03 13:17:03 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks resource profile 0
25/01/03 13:17:03 INFO StandaloneSchedulerBackend$StandaloneDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.86.198:55061) with ID 0, ResourceProfileId 0
25/01/03 13:17:03 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.86.198:55063 with 434.4 MiB RAM, BlockManagerId(0, 192.168.86.198, 55063, None)
25/01/03 13:17:05 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 10914 bytes)
25/01/03 13:17:05 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1) (192.168.86.198, executor 0, partition 1, PROCESS_LOCAL, 10914 bytes)
25/01/03 13:17:05 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.86.198:55063 (size: 69.0 KiB, free: 434.3 MiB)
25/01/03 13:17:10 INFO BlockManagerInfo: Added rdd_13_1 in memory on 192.168.86.198:55063 (size: 616.0 B, free: 434.3 MiB)
25/01/03 13:17:10 INFO BlockManagerInfo: Added rdd_13_0 in memory on 192.168.86.198:55063 (size: 616.0 B, free: 434.3 MiB)
25/01/03 13:17:10 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 5259 ms on 192.168.86.198 (executor 0) (1/2)
25/01/03 13:17:10 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 5267 ms on 192.168.86.198 (executor 0) (2/2)
25/01/03 13:17:10 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
25/01/03 13:17:10 INFO DAGScheduler: ShuffleMapStage 0 (mapToPair at GroupNonMergingWindowsFunctions.java:273) finished in 7,352 s
25/01/03 13:17:10 INFO DAGScheduler: looking for newly runnable stages
25/01/03 13:17:10 INFO DAGScheduler: running: Set()
25/01/03 13:17:10 INFO DAGScheduler: waiting: Set(ShuffleMapStage 1, ResultStage 2)
25/01/03 13:17:10 INFO DAGScheduler: failed: Set()
25/01/03 13:17:10 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[37] at repartition at GroupCombineFunctions.java:191), which has no missing parents
25/01/03 13:17:10 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 225.4 KiB, free 433.9 MiB)
25/01/03 13:17:10 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 72.3 KiB, free 433.8 MiB)
25/01/03 13:17:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.86.198:55059 (size: 72.3 KiB, free: 434.3 MiB)
25/01/03 13:17:10 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1585
25/01/03 13:17:10 INFO DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[37] at repartition at GroupCombineFunctions.java:191) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
25/01/03 13:17:10 INFO TaskSchedulerImpl: Adding task set 1.0 with 4 tasks resource profile 0
25/01/03 13:17:10 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 11023 bytes)
25/01/03 13:17:10 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3) (192.168.86.198, executor 0, partition 1, PROCESS_LOCAL, 11023 bytes)
25/01/03 13:17:10 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 4) (192.168.86.198, executor 0, partition 2, PROCESS_LOCAL, 9091 bytes)
25/01/03 13:17:10 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 5) (192.168.86.198, executor 0, partition 3, PROCESS_LOCAL, 9091 bytes)
25/01/03 13:17:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.86.198:55063 (size: 72.3 KiB, free: 434.3 MiB)
25/01/03 13:17:10 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 192.168.86.198:55061
25/01/03 13:17:10 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 85 ms on 192.168.86.198 (executor 0) (1/4)
25/01/03 13:17:10 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 84 ms on 192.168.86.198 (executor 0) (2/4)
25/01/03 13:17:10 INFO BlockManagerInfo: Added rdd_24_1 in memory on 192.168.86.198:55063 (size: 16.0 B, free: 434.3 MiB)
25/01/03 13:17:10 INFO BlockManagerInfo: Added rdd_24_0 in memory on 192.168.86.198:55063 (size: 16.0 B, free: 434.3 MiB)
25/01/03 13:17:10 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 5) in 499 ms on 192.168.86.198 (executor 0) (3/4)
25/01/03 13:17:10 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 4) in 499 ms on 192.168.86.198 (executor 0) (4/4)
25/01/03 13:17:10 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
25/01/03 13:17:10 INFO DAGScheduler: ShuffleMapStage 1 (repartition at GroupCombineFunctions.java:191) finished in 0,514 s
25/01/03 13:17:10 INFO DAGScheduler: looking for newly runnable stages
25/01/03 13:17:10 INFO DAGScheduler: running: Set()
25/01/03 13:17:10 INFO DAGScheduler: waiting: Set(ResultStage 2)
25/01/03 13:17:10 INFO DAGScheduler: failed: Set()
25/01/03 13:17:10 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[48] at map at BoundedDataset.java:95), which has no missing parents
25/01/03 13:17:10 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 21.4 KiB, free 433.8 MiB)
25/01/03 13:17:10 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 9.0 KiB, free 433.8 MiB)
25/01/03 13:17:10 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.86.198:55059 (size: 9.0 KiB, free: 434.3 MiB)
25/01/03 13:17:10 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1585
25/01/03 13:17:10 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 2 (MapPartitionsRDD[48] at map at BoundedDataset.java:95) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
25/01/03 13:17:10 INFO TaskSchedulerImpl: Adding task set 2.0 with 4 tasks resource profile 0
25/01/03 13:17:10 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 6) (192.168.86.198, executor 0, partition 1, NODE_LOCAL, 9269 bytes)
25/01/03 13:17:10 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 7) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:10 INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 8) (192.168.86.198, executor 0, partition 2, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:10 INFO TaskSetManager: Starting task 3.0 in stage 2.0 (TID 9) (192.168.86.198, executor 0, partition 3, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:10 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.86.198:55063 (size: 9.0 KiB, free: 434.3 MiB)
25/01/03 13:17:11 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.168.86.198:55061
25/01/03 13:17:11 INFO BlockManagerInfo: Added rdd_44_0 in memory on 192.168.86.198:55063 (size: 16.0 B, free: 434.3 MiB)
25/01/03 13:17:11 INFO BlockManagerInfo: Added rdd_44_3 in memory on 192.168.86.198:55063 (size: 16.0 B, free: 434.3 MiB)
25/01/03 13:17:11 INFO BlockManagerInfo: Added rdd_44_2 in memory on 192.168.86.198:55063 (size: 16.0 B, free: 434.3 MiB)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 3.0 in stage 2.0 (TID 9) in 62 ms on 192.168.86.198 (executor 0) (1/4)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 2.0 in stage 2.0 (TID 8) in 62 ms on 192.168.86.198 (executor 0) (2/4)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 7) in 63 ms on 192.168.86.198 (executor 0) (3/4)
25/01/03 13:17:11 INFO BlockManagerInfo: Added rdd_44_1 in memory on 192.168.86.198:55063 (size: 808.0 B, free: 434.3 MiB)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 6) in 80 ms on 192.168.86.198 (executor 0) (4/4)
25/01/03 13:17:11 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
25/01/03 13:17:11 INFO DAGScheduler: ResultStage 2 (collect at BoundedDataset.java:96) finished in 0,087 s
25/01/03 13:17:11 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
25/01/03 13:17:11 INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished
25/01/03 13:17:11 INFO DAGScheduler: Job 0 finished: collect at BoundedDataset.java:96, took 8,010176 s
25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating Impulse
25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.MapElements$2@685e8e17
25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.Reify$ReifyView$1@5d6e77a4
25/01/03 13:17:11 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 448.0 B, free 433.8 MiB)
25/01/03 13:17:11 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 330.0 B, free 433.8 MiB)
25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.86.198:55059 (size: 330.0 B, free: 434.3 MiB)
25/01/03 13:17:11 INFO SparkContext: Created broadcast 3 from broadcast at SideInputBroadcast.java:62
25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.MapElements$2@72139933
25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.MapElements$2@4f0b02a3
25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn@280d1d8d
25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.Reshuffle$AssignShardFn@5abfb698
25/01/03 13:17:11 INFO SparkRunner$Evaluator: Entering directly-translatable composite transform: 'BeamOutputTransform/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle'
25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating Reshuffle
25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.MapElements$2@64a0a1c6
25/01/03 13:17:11 INFO SparkContext: Starting job: foreach at BoundedDataset.java:127
25/01/03 13:17:11 INFO DAGScheduler: Got job 1 (foreach at BoundedDataset.java:127) with 2 output partitions
25/01/03 13:17:11 INFO DAGScheduler: Final stage: ResultStage 3 (foreach at BoundedDataset.java:127)
25/01/03 13:17:11 INFO DAGScheduler: Parents of final stage: List()
25/01/03 13:17:11 INFO DAGScheduler: Missing parents: List()
25/01/03 13:17:11 INFO DAGScheduler: Submitting ResultStage 3 (BeamOutputTransform/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles.out2 MapPartitionsRDD[19] at values at TransformTranslator.java:472), which has no missing parents
25/01/03 13:17:11 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 213.7 KiB, free 433.6 MiB)
25/01/03 13:17:11 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 68.1 KiB, free 433.5 MiB)
25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.86.198:55059 (size: 68.1 KiB, free: 434.2 MiB)
25/01/03 13:17:11 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1585
25/01/03 13:17:11 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 3 (BeamOutputTransform/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles.out2 MapPartitionsRDD[19] at values at TransformTranslator.java:472) (first 15 tasks are for partitions Vector(0, 1))
25/01/03 13:17:11 INFO TaskSchedulerImpl: Adding task set 3.0 with 2 tasks resource profile 0
25/01/03 13:17:11 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 10) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 10925 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 11) (192.168.86.198, executor 0, partition 1, PROCESS_LOCAL, 10925 bytes)
25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.86.198:55063 (size: 68.1 KiB, free: 434.2 MiB)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 11) in 27 ms on 192.168.86.198 (executor 0) (1/2)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 10) in 28 ms on 192.168.86.198 (executor 0) (2/2)
25/01/03 13:17:11 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
25/01/03 13:17:11 INFO DAGScheduler: ResultStage 3 (foreach at BoundedDataset.java:127) finished in 0,033 s
25/01/03 13:17:11 INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job
25/01/03 13:17:11 INFO TaskSchedulerImpl: Killing all running tasks in stage 3: Stage finished
25/01/03 13:17:11 INFO DAGScheduler: Job 1 finished: foreach at BoundedDataset.java:127, took 0,035680 s
25/01/03 13:17:11 INFO SparkContext: Starting job: foreach at BoundedDataset.java:127
25/01/03 13:17:11 INFO DAGScheduler: Got job 2 (foreach at BoundedDataset.java:127) with 2 output partitions
25/01/03 13:17:11 INFO DAGScheduler: Final stage: ResultStage 5 (foreach at BoundedDataset.java:127)
25/01/03 13:17:11 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 4)
25/01/03 13:17:11 INFO DAGScheduler: Missing parents: List()
25/01/03 13:17:11 INFO DAGScheduler: Submitting ResultStage 5 (BeamOutputTransform/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten.out1 MapPartitionsRDD[28] at values at TransformTranslator.java:472), which has no missing parents
25/01/03 13:17:11 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 221.9 KiB, free 433.3 MiB)
25/01/03 13:17:11 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 71.3 KiB, free 433.2 MiB)
25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.86.198:55059 (size: 71.3 KiB, free: 434.1 MiB)
25/01/03 13:17:11 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1585
25/01/03 13:17:11 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 5 (BeamOutputTransform/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten.out1 MapPartitionsRDD[28] at values at TransformTranslator.java:472) (first 15 tasks are for partitions Vector(0, 1))
25/01/03 13:17:11 INFO TaskSchedulerImpl: Adding task set 5.0 with 2 tasks resource profile 0
25/01/03 13:17:11 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 12) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 8993 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID 13) (192.168.86.198, executor 0, partition 1, PROCESS_LOCAL, 8993 bytes)
25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.86.198:55063 (size: 71.3 KiB, free: 434.1 MiB)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 1.0 in stage 5.0 (TID 13) in 18 ms on 192.168.86.198 (executor 0) (1/2)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 12) in 18 ms on 192.168.86.198 (executor 0) (2/2)
25/01/03 13:17:11 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool
25/01/03 13:17:11 INFO DAGScheduler: ResultStage 5 (foreach at BoundedDataset.java:127) finished in 0,024 s
25/01/03 13:17:11 INFO DAGScheduler: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job
25/01/03 13:17:11 INFO TaskSchedulerImpl: Killing all running tasks in stage 5: Stage finished
25/01/03 13:17:11 INFO DAGScheduler: Job 2 finished: foreach at BoundedDataset.java:127, took 0,027143 s
25/01/03 13:17:11 INFO SparkContext: Starting job: foreach at BoundedDataset.java:127
25/01/03 13:17:11 INFO DAGScheduler: Got job 3 (foreach at BoundedDataset.java:127) with 4 output partitions
25/01/03 13:17:11 INFO DAGScheduler: Final stage: ResultStage 8 (foreach at BoundedDataset.java:127)
25/01/03 13:17:11 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 7)
25/01/03 13:17:11 INFO DAGScheduler: Missing parents: List()
25/01/03 13:17:11 INFO DAGScheduler: Submitting ResultStage 8 (BeamOutputTransform/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsIterable/MapElements/Map/ParMultiDo(Anonymous).output MapPartitionsRDD[47] at values at TransformTranslator.java:472), which has no missing parents
25/01/03 13:17:11 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 22.5 KiB, free 433.2 MiB)
25/01/03 13:17:11 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 9.4 KiB, free 433.2 MiB)
25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 192.168.86.198:55059 (size: 9.4 KiB, free: 434.1 MiB)
25/01/03 13:17:11 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1585
25/01/03 13:17:11 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 8 (BeamOutputTransform/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsIterable/MapElements/Map/ParMultiDo(Anonymous).output MapPartitionsRDD[47] at values at TransformTranslator.java:472) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
25/01/03 13:17:11 INFO TaskSchedulerImpl: Adding task set 8.0 with 4 tasks resource profile 0
25/01/03 13:17:11 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 14) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 1.0 in stage 8.0 (TID 15) (192.168.86.198, executor 0, partition 1, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 2.0 in stage 8.0 (TID 16) (192.168.86.198, executor 0, partition 2, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 3.0 in stage 8.0 (TID 17) (192.168.86.198, executor 0, partition 3, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 192.168.86.198:55063 (size: 9.4 KiB, free: 434.1 MiB)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 2.0 in stage 8.0 (TID 16) in 17 ms on 192.168.86.198 (executor 0) (1/4)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 14) in 19 ms on 192.168.86.198 (executor 0) (2/4)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 3.0 in stage 8.0 (TID 17) in 18 ms on 192.168.86.198 (executor 0) (3/4)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 1.0 in stage 8.0 (TID 15) in 18 ms on 192.168.86.198 (executor 0) (4/4)
25/01/03 13:17:11 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool
25/01/03 13:17:11 INFO DAGScheduler: ResultStage 8 (foreach at BoundedDataset.java:127) finished in 0,023 s
25/01/03 13:17:11 INFO DAGScheduler: Job 3 is finished. Cancelling potential speculative or zombie tasks for this job
25/01/03 13:17:11 INFO TaskSchedulerImpl: Killing all running tasks in stage 8: Stage finished
25/01/03 13:17:11 INFO DAGScheduler: Job 3 finished: foreach at BoundedDataset.java:127, took 0,027300 s
25/01/03 13:17:11 INFO SparkContext: Starting job: foreach at BoundedDataset.java:127
25/01/03 13:17:11 INFO DAGScheduler: Registering RDD 70 (repartition at GroupCombineFunctions.java:191) as input to shuffle 2
25/01/03 13:17:11 INFO DAGScheduler: Got job 4 (foreach at BoundedDataset.java:127) with 12 output partitions
25/01/03 13:17:11 INFO DAGScheduler: Final stage: ResultStage 10 (foreach at BoundedDataset.java:127)
25/01/03 13:17:11 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 9)
25/01/03 13:17:11 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 9)
25/01/03 13:17:11 INFO DAGScheduler: Submitting ShuffleMapStage 9 (MapPartitionsRDD[70] at repartition at GroupCombineFunctions.java:191), which has no missing parents
25/01/03 13:17:11 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 32.5 KiB, free 433.2 MiB)
25/01/03 13:17:11 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 13.1 KiB, free 433.2 MiB)
25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on 192.168.86.198:55059 (size: 13.1 KiB, free: 434.1 MiB)
25/01/03 13:17:11 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1585
25/01/03 13:17:11 INFO DAGScheduler: Submitting 12 missing tasks from ShuffleMapStage 9 (MapPartitionsRDD[70] at repartition at GroupCombineFunctions.java:191) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11))
25/01/03 13:17:11 INFO TaskSchedulerImpl: Adding task set 9.0 with 12 tasks resource profile 0
25/01/03 13:17:11 INFO TaskSetManager: Starting task 0.0 in stage 9.0 (TID 18) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 9160 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 1.0 in stage 9.0 (TID 19) (192.168.86.198, executor 0, partition 1, PROCESS_LOCAL, 9160 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 2.0 in stage 9.0 (TID 20) (192.168.86.198, executor 0, partition 2, PROCESS_LOCAL, 9160 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 3.0 in stage 9.0 (TID 21) (192.168.86.198, executor 0, partition 3, PROCESS_LOCAL, 9160 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 4.0 in stage 9.0 (TID 22) (192.168.86.198, executor 0, partition 4, PROCESS_LOCAL, 9160 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 5.0 in stage 9.0 (TID 23) (192.168.86.198, executor 0, partition 5, PROCESS_LOCAL, 9160 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 6.0 in stage 9.0 (TID 24) (192.168.86.198, executor 0, partition 6, PROCESS_LOCAL, 9160 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 7.0 in stage 9.0 (TID 25) (192.168.86.198, executor 0, partition 7, PROCESS_LOCAL, 9160 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 8.0 in stage 9.0 (TID 26) (192.168.86.198, executor 0, partition 8, PROCESS_LOCAL, 9160 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 9.0 in stage 9.0 (TID 27) (192.168.86.198, executor 0, partition 9, PROCESS_LOCAL, 9160 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 10.0 in stage 9.0 (TID 28) (192.168.86.198, executor 0, partition 10, PROCESS_LOCAL, 9160 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 11.0 in stage 9.0 (TID 29) (192.168.86.198, executor 0, partition 11, PROCESS_LOCAL, 9171 bytes)
25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on 192.168.86.198:55063 (size: 13.1 KiB, free: 434.1 MiB)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 3.0 in stage 9.0 (TID 21) in 52 ms on 192.168.86.198 (executor 0) (1/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 10.0 in stage 9.0 (TID 28) in 50 ms on 192.168.86.198 (executor 0) (2/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 5.0 in stage 9.0 (TID 23) in 53 ms on 192.168.86.198 (executor 0) (3/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 9.0 in stage 9.0 (TID 27) in 51 ms on 192.168.86.198 (executor 0) (4/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 2.0 in stage 9.0 (TID 20) in 53 ms on 192.168.86.198 (executor 0) (5/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 8.0 in stage 9.0 (TID 26) in 51 ms on 192.168.86.198 (executor 0) (6/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 1.0 in stage 9.0 (TID 19) in 54 ms on 192.168.86.198 (executor 0) (7/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 7.0 in stage 9.0 (TID 25) in 53 ms on 192.168.86.198 (executor 0) (8/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 6.0 in stage 9.0 (TID 24) in 55 ms on 192.168.86.198 (executor 0) (9/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 4.0 in stage 9.0 (TID 22) in 55 ms on 192.168.86.198 (executor 0) (10/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 0.0 in stage 9.0 (TID 18) in 57 ms on 192.168.86.198 (executor 0) (11/12)
25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.86.198:55063 (size: 330.0 B, free: 434.1 MiB)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 11.0 in stage 9.0 (TID 29) in 107 ms on 192.168.86.198 (executor 0) (12/12)
25/01/03 13:17:11 INFO TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool
25/01/03 13:17:11 INFO DAGScheduler: ShuffleMapStage 9 (repartition at GroupCombineFunctions.java:191) finished in 0,115 s
25/01/03 13:17:11 INFO DAGScheduler: looking for newly runnable stages
25/01/03 13:17:11 INFO DAGScheduler: running: Set()
25/01/03 13:17:11 INFO DAGScheduler: waiting: Set(ResultStage 10)
25/01/03 13:17:11 INFO DAGScheduler: failed: Set()
25/01/03 13:17:11 INFO DAGScheduler: Submitting ResultStage 10 (BeamOutputTransform/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map/ParMultiDo(Anonymous).output MapPartitionsRDD[77] at values at TransformTranslator.java:472), which has no missing parents
25/01/03 13:17:11 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 21.2 KiB, free 433.1 MiB)
25/01/03 13:17:11 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 8.9 KiB, free 433.1 MiB)
25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on 192.168.86.198:55059 (size: 8.9 KiB, free: 434.1 MiB)
25/01/03 13:17:11 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1585
25/01/03 13:17:11 INFO DAGScheduler: Submitting 12 missing tasks from ResultStage 10 (BeamOutputTransform/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map/ParMultiDo(Anonymous).output MapPartitionsRDD[77] at values at TransformTranslator.java:472) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11))
25/01/03 13:17:11 INFO TaskSchedulerImpl: Adding task set 10.0 with 12 tasks resource profile 0
25/01/03 13:17:11 INFO TaskSetManager: Starting task 3.0 in stage 10.0 (TID 30) (192.168.86.198, executor 0, partition 3, NODE_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 4.0 in stage 10.0 (TID 31) (192.168.86.198, executor 0, partition 4, NODE_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 0.0 in stage 10.0 (TID 32) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 1.0 in stage 10.0 (TID 33) (192.168.86.198, executor 0, partition 1, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 2.0 in stage 10.0 (TID 34) (192.168.86.198, executor 0, partition 2, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 5.0 in stage 10.0 (TID 35) (192.168.86.198, executor 0, partition 5, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 6.0 in stage 10.0 (TID 36) (192.168.86.198, executor 0, partition 6, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 7.0 in stage 10.0 (TID 37) (192.168.86.198, executor 0, partition 7, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 8.0 in stage 10.0 (TID 38) (192.168.86.198, executor 0, partition 8, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 9.0 in stage 10.0 (TID 39) (192.168.86.198, executor 0, partition 9, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 10.0 in stage 10.0 (TID 40) (192.168.86.198, executor 0, partition 10, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO TaskSetManager: Starting task 11.0 in stage 10.0 (TID 41) (192.168.86.198, executor 0, partition 11, PROCESS_LOCAL, 9269 bytes)
25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on 192.168.86.198:55063 (size: 8.9 KiB, free: 434.1 MiB)
25/01/03 13:17:11 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 192.168.86.198:55061
25/01/03 13:17:11 INFO TaskSetManager: Finished task 6.0 in stage 10.0 (TID 36) in 21 ms on 192.168.86.198 (executor 0) (1/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 3.0 in stage 10.0 (TID 30) in 23 ms on 192.168.86.198 (executor 0) (2/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 5.0 in stage 10.0 (TID 35) in 22 ms on 192.168.86.198 (executor 0) (3/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 1.0 in stage 10.0 (TID 33) in 23 ms on 192.168.86.198 (executor 0) (4/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 10.0 in stage 10.0 (TID 40) in 21 ms on 192.168.86.198 (executor 0) (5/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 0.0 in stage 10.0 (TID 32) in 24 ms on 192.168.86.198 (executor 0) (6/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 11.0 in stage 10.0 (TID 41) in 22 ms on 192.168.86.198 (executor 0) (7/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 2.0 in stage 10.0 (TID 34) in 24 ms on 192.168.86.198 (executor 0) (8/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 9.0 in stage 10.0 (TID 39) in 24 ms on 192.168.86.198 (executor 0) (9/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 8.0 in stage 10.0 (TID 38) in 24 ms on 192.168.86.198 (executor 0) (10/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 7.0 in stage 10.0 (TID 37) in 24 ms on 192.168.86.198 (executor 0) (11/12)
25/01/03 13:17:11 INFO TaskSetManager: Finished task 4.0 in stage 10.0 (TID 31) in 25 ms on 192.168.86.198 (executor 0) (12/12)
25/01/03 13:17:11 INFO TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool
25/01/03 13:17:11 INFO DAGScheduler: ResultStage 10 (foreach at BoundedDataset.java:127) finished in 0,030 s
25/01/03 13:17:11 INFO DAGScheduler: Job 4 is finished. Cancelling potential speculative or zombie tasks for this job
25/01/03 13:17:11 INFO TaskSchedulerImpl: Killing all running tasks in stage 10: Stage finished
25/01/03 13:17:11 INFO DAGScheduler: Job 4 finished: foreach at BoundedDataset.java:127, took 0,152777 s
25/01/03 13:17:11 INFO SparkRunner: Batch pipeline execution complete.
25/01/03 13:17:11 INFO SparkContext: SparkContext is stopping with exitCode 0.
25/01/03 13:17:11 INFO SparkUI: Stopped Spark web UI at http://192.168.86.198:4040
25/01/03 13:17:11 INFO StandaloneSchedulerBackend: Shutting down all executors
25/01/03 13:17:11 INFO StandaloneSchedulerBackend$StandaloneDriverEndpoint: Asking each executor to shut down
25/01/03 13:17:11 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
25/01/03 13:17:11 INFO MemoryStore: MemoryStore cleared
25/01/03 13:17:11 INFO BlockManager: BlockManager stopped
25/01/03 13:17:11 INFO BlockManagerMaster: BlockManagerMaster stopped
25/01/03 13:17:11 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
25/01/03 13:17:11 INFO SparkContext: Successfully stopped SparkContext
2025/01/03 13:17:11 - General - Beam pipeline execution has finished.
>>>>>> Execution finished...
25/01/03 13:17:11 INFO ShutdownHookManager: Shutdown hook called
25/01/03 13:17:11 INFO ShutdownHookManager: Deleting directory /private/var/folders/v0/hjm6kpgd0076ncdkpf8rg3dw0000gn/T/spark-5a592189-8383-44ae-8574-70c2aaabba19
25/01/03 13:17:11 INFO ShutdownHookManager: Deleting directory /private/var/folders/v0/hjm6kpgd0076ncdkpf8rg3dw0000gn/T/spark-611dd39c-8451-4aed-9c91-ea906a4e9e04
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "Raja10D (via GitHub)" <gi...@apache.org>.
Raja10D commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2572428433
![Uploading Screenshot from 2025-01-06 12-31-38.png…]()
If I am running my pipeline Hop-Gui itself it is causing error with Beam Spark Pipeline Engine
**![Uploading Screenshot from 2025-01-06 12-31-53.png…]()**
**Error in Hop-Gui:**
2025/01/06 12:17:47 - Hop - Pipeline opened.
2025/01/06 12:17:47 - Hop - Launching pipeline [hop_trans2]...
2025/01/06 12:17:47 - Hop - Started the pipeline execution.
2025/01/06 12:17:53 - Hop - ERROR: hop_trans2: preparing pipeline execution failed
2025/01/06 12:17:53 - Hop - ERROR: org.apache.hop.core.exception.HopException:
2025/01/06 12:17:53 - Hop - Error preparing remote pipeline
2025/01/06 12:17:53 - Hop - Error converting Hop pipeline to Beam
2025/01/06 12:17:53 - Hop -
2025/01/06 12:17:53 - Hop - at org.apache.hop.beam.engines.BeamPipelineEngine.prepareExecution(BeamPipelineEngine.java:296)
2025/01/06 12:17:53 - Hop - at org.apache.hop.ui.hopgui.file.pipeline.HopGuiPipelineGraph.lambda$preparePipeline$15(HopGuiPipelineGraph.java:4532)
2025/01/06 12:17:53 - Hop - at java.base/java.lang.Thread.run(Thread.java:840)
2025/01/06 12:17:53 - Hop - Caused by: java.lang.Exception: Error converting Hop pipeline to Beam
2025/01/06 12:17:53 - Hop - at org.apache.hop.beam.pipeline.HopPipelineMetaToBeamPipelineConverter.createPipeline(HopPipelineMetaToBeamPipelineConverter.java:308)
2025/01/06 12:17:53 - Hop - at org.apache.hop.beam.engines.BeamPipelineEngine.prepareExecution(BeamPipelineEngine.java:265)
2025/01/06 12:17:53 - Hop - ... 2 more
2025/01/06 12:17:53 - Hop - Caused by: java.lang.NoClassDefFoundError: org/apache/spark/metrics/source/Source
2025/01/06 12:17:53 - Hop - at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
2025/01/06 12:17:53 - Hop - at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3402)
2025/01/06 12:17:53 - Hop - at java.base/java.lang.Class.getDeclaredMethod(Class.java:2673)
2025/01/06 12:17:53 - Hop - at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:194)
2025/01/06 12:17:53 - Hop - at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:158)
2025/01/06 12:17:53 - Hop - at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
2025/01/06 12:17:53 - Hop - at org.apache.beam.sdk.Pipeline.create(Pipeline.java:158)
2025/01/06 12:17:53 - Hop - at org.apache.hop.beam.pipeline.HopPipelineMetaToBeamPipelineConverter.createPipeline(HopPipelineMetaToBeamPipelineConverter.java:283)
2025/01/06 12:17:53 - Hop - ... 3 more
2025/01/06 12:17:53 - Hop - Caused by: java.lang.ClassNotFoundException: org.apache.spark.metrics.source.Source
2025/01/06 12:17:53 - Hop - at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
2025/01/06 12:17:53 - Hop - at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
2025/01/06 12:17:53 - Hop - at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
2025/01/06 12:17:53 - Hop - ... 11 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2569080538
Have you tried the steps from our guide and does it give this error?
https://hop.apache.org//manual/latest/pipeline/beam/beam-samples-spark.html#_get_spark
It seems like a jar might be missing. But it could also be a mismatch on spark/scala versions between Hop and your cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "Raja10D (via GitHub)" <gi...@apache.org>.
Raja10D commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2569133590
decoders@decoders-Latitude-7480:~/Downloads/apache-hop-client-2.11.0/hop$ spark-submit \
--class org.apache.hop.beam.run.MainBeam \
--master local[4] \
--jars /home/decoders/Downloads/apache-hop-client-2.11.0/hop/lib/beam/beam-runners-spark-3-2.61.0.jar \
/home/decoders/Downloads/apache-hop-client-2.11.0/fatjar.jar \
--runner=SparkRunner \
--sparkMaster=local[4] \
--tempLocation=/home/decoders/custom/temp \
--pipeline=/home/decoders/Downloads/hop_trans2.hpl \
--config=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/hop-config.json
25/01/03 17:26:34 WARN Utils: Your hostname, decoders-Latitude-7480 resolves to a loopback address: 127.0.1.1; using 192.125.1.140 instead (on interface enp0s31f6)
25/01/03 17:26:34 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
25/01/03 17:26:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>>>>>> Initializing Hop
Error running Beam pipeline:
Error reading metadata from JSON
Error find metadata class for key null
The metadata plugin for key null could not be found in the plugin registry
org.apache.hop.core.exception.HopException:
Error reading metadata from JSON
Error find metadata class for key null
The metadata plugin for key null could not be found in the plugin registry
at org.apache.hop.core.metadata.SerializableMetadataProvider.<init>(SerializableMetadataProvider.java:143)
at org.apache.hop.beam.run.MainBeam.main(MainBeam.java:101)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1034)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hop.core.exception.HopException:
Error find metadata class for key null
The metadata plugin for key null could not be found in the plugin registry
at org.apache.hop.metadata.serializer.BaseMetadataProvider.getMetadataClassForKey(BaseMetadataProvider.java:69)
at org.apache.hop.core.metadata.SerializableMetadataProvider.<init>(SerializableMetadataProvider.java:122)
... 13 more
Caused by: org.apache.hop.core.exception.HopException:
The metadata plugin for key null could not be found in the plugin registry
at org.apache.hop.metadata.serializer.BaseMetadataProvider.getMetadataClassForKey(BaseMetadataProvider.java:61)
... 14 more
25/01/03 17:26:51 INFO ShutdownHookManager: Shutdown hook called
25/01/03 17:26:51 INFO ShutdownHookManager: Deleting directory /tmp/spark-f18e078a-acef-43c4-9b67-72880678ee49
decoders@decoders-Latitude-7480:~/Downloads/apache-hop-client-2.11.0/hop$
**This is the issue we are facing. The documentation you shared is that we are following**
**Any ways to debug this issue
Can you give suggestions for running on Spark**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2572949631
good, but this still looks like a configuration issue. try following **all** of the steps in the [Beam + Spark](https://hop.apache.org/manual/latest/pipeline/beam/beam-samples-spark.html) guide.
If you can get the sample pipeline to work, you'll know your configuration is fine, that will make troubleshooting your own pipelines a lot easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2572861636
`25/01/06 15:55:27 WARN Utils: Your hostname, decoders-Latitude-7480 resolves to a loopback address: 127.0.1.1; using 192.125.1.140 instead (on interface enp0s31f6)`
this looks like an error in your spark configuration. can you run any of the spark sample commands (word count etc) using spark-submit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "Raja10D (via GitHub)" <gi...@apache.org>.
Raja10D commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2574571300
Ok, Thanks for all your replies.
I can't get the output till now.
If you upload videos of running hop pipelines with Beam Spark Runner with latest versions, it will be very helpful.
I have seen the videos and followed the video also which is 4 years old, that is not working with latest versions.
Last one final questions can you tell what all the versions I can use for my pipeline(inserted the image below) to run and what all the jars I need to download seperately for my pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "Raja10D (via GitHub)" <gi...@apache.org>.
Raja10D commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2569090568
Apache Beam Version: 2.61.0
Apache Spark Version: 3.5.1
Apache Hop Version: apache-hop-client-2.11.0
These are the versions we are using
Can you help me what all the versions I have to be use
Our team is working on more than 10days, We are following the same
documentation.
On Fri, Jan 3, 2025 at 4:58 PM Hans Van Akelyen ***@***.***>
wrote:
> Have you tried the steps from our guide and does it give this error?
>
>
> https://hop.apache.org//manual/latest/pipeline/beam/beam-samples-spark.html#_get_spark
>
> It seems like a jar might be missing. But it could also be a mismatch on
> spark/scala versions between Hop and your cluster
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/hop/issues/4752#issuecomment-2569080538>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BOEJ2FAL7ZZEZDYY4OQPPTL2IZX6HAVCNFSM6AAAAABUQACHGKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRZGA4DANJTHA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2572449621
why are you adding `--pipeline` and `--metadata`? where did you get that from? It's not in my example and I don't think it's in the docs...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2569147431
make sure to also follow the steps [here](https://hop.apache.org//manual/latest/pipeline/beam/running-the-beam-samples.html) your metadata.json is wrong. It are also arguments to the fat-jar so no need to add `--pipeline` and things like that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2569217686
The metadata is wrong you need to export it from the client it is not the same as the project-config.json as explained in the second link I sent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "Raja10D (via GitHub)" <gi...@apache.org>.
Raja10D commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2572912151
25/01/06 16:53:23 INFO DAGScheduler: Job 0 finished: collect at /home/decoders/Downloads/spark-3.5.4-bin-hadoop3/examples/src/main/python/wordcount.py:38, took 3.844584 s
Michael,: 1
29: 1
Andy,: 1
30: 1
Justin,: 1
19: 1
25/01/06 16:53:23 INFO SparkContext: SparkContext is stopping with exitCode 0.
25/01/06 16:53:23 INFO SparkUI: Stopped Spark web UI at http://192.125.1.140:4040
25/01/06 16:53:23 INFO StandaloneSchedulerBackend: Shutting down all executors
25/01/06 16:53:23 INFO StandaloneSchedulerBackend$StandaloneDriverEndpoint: Asking each executor to shut down
25/01/06 16:53:23 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
25/01/06 16:53:23 INFO MemoryStore: MemoryStore cleared
25/01/06 16:53:23 INFO BlockManager: BlockManager stopped
25/01/06 16:53:23 INFO BlockManagerMaster: BlockManagerMaster stopped
25/01/06 16:53:23 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
25/01/06 16:53:23 INFO SparkContext: Successfully stopped SparkContext
25/01/06 16:53:24 INFO ShutdownHookManager: Shutdown hook called
25/01/06 16:53:24 INFO ShutdownHookManager: Deleting directory /tmp/spark-e8385fb4-8098-4e0a-96ad-f0c90d8ae702/pyspark-bdad7648-0094-4df7-bf3d-f4251143f648
25/01/06 16:53:24 INFO ShutdownHookManager: Deleting directory /tmp/spark-3699bf1c-f16a-41f5-be16-e5311c9d55ed
25/01/06 16:53:24 INFO ShutdownHookManager: Deleting directory /tmp/spark-e8385fb4-8098-4e0a-96ad-f0c90d8ae702
spark-submit word-count program runs successfully
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "Raja10D (via GitHub)" <gi...@apache.org>.
Raja10D commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2572816889
**Thanks for the reply, I've changed the command, now also recieving error**
Command:
bin/spark-submit \
--master spark://decoders-Latitude-7480:7077 \
--class org.apache.hop.beam.run.MainBeam \
--driver-java-options '-DPROJECT_HOME=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default' \
/home/decoders/Downloads/apache-hop-client-2.11.0/fatjar.jar \
/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/pipelines/hop_trans2.hpl \
/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/metadata.json \
Beam_Spark_pipeline_engine
Error:
decoders@decoders-Latitude-7480:~/Downloads/spark-3.5.4-bin-hadoop3$ bin/spark-submit \
--master spark://decoders-Latitude-7480:7077 \
--class org.apache.hop.beam.run.MainBeam \
--driver-java-options '-DPROJECT_HOME=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default' \
/home/decoders/Downloads/apache-hop-client-2.11.0/fatjar.jar \
/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/pipelines/hop_trans2.hpl \
/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/metadata.json \
Beam_Spark_pipeline_engine
25/01/06 15:55:27 WARN Utils: Your hostname, decoders-Latitude-7480 resolves to a loopback address: 127.0.1.1; using 192.125.1.140 instead (on interface enp0s31f6)
25/01/06 15:55:27 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
>>>>>> Initializing Hop
Hop configuration file not found, not serializing: /home/decoders/Downloads/spark-3.5.4-bin-hadoop3/config/hop-config.json
Argument 1 : Pipeline filename (.hpl) : /home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/pipelines/hop_trans2.hpl
Argument 2 : Environment state filename: (.json) : /home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/metadata.json
Argument 3 : Pipeline run configuration : Beam_Spark_pipeline_engine
>>>>>> Loading pipeline metadata
>>>>>> Building Apache Beam Pipeline...
>>>>>> Pipeline executing starting...
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/beam/sdk/options/PipelineOptions
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:579)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
at org.apache.hop.beam.engines.spark.BeamSparkPipelineRunConfiguration.getPipelineOptions(BeamSparkPipelineRunConfiguration.java:213)
at org.apache.hop.beam.pipeline.HopPipelineMetaToBeamPipelineConverter.<init>(HopPipelineMetaToBeamPipelineConverter.java:119)
at org.apache.hop.beam.engines.BeamPipelineEngine.prepareExecution(BeamPipelineEngine.java:263)
at org.apache.hop.beam.engines.BeamPipelineEngine.execute(BeamPipelineEngine.java:633)
at org.apache.hop.beam.run.MainBeam.main(MainBeam.java:152)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1034)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.options.PipelineOptions
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
... 36 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva closed issue #4752: [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine
URL: https://github.com/apache/hop/issues/4752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "Raja10D (via GitHub)" <gi...@apache.org>.
Raja10D commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2572447107
> as mentioned [here](https://hop.apache.org/manual/latest/pipeline/beam/beam-samples-spark.html#_run_sample_pipeline_with_spark_submit): `Since Spark doesn’t support remote execution, we’ll be running one of the sample pipelines through Spark Submit.` "Spark doesn't support remote execution" basically means you can't run from Hop Gui (there could be ways to make it work, but those would probably not production-like). We'll see if we can explain this better in the docs. Can you try your sample again with `spark-submit` like Hans showed and let us know if that helps?
Ok, understand
But With Spark-Submit also our team is facing same error
Whole Team is facing the same error
The error I mentioned in the above comment before the comment you replied
Please see to that and give us solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2572437336
as mentioned [here](https://hop.apache.org/manual/latest/pipeline/beam/beam-samples-spark.html#_run_sample_pipeline_with_spark_submit): `Since Spark doesn’t support remote execution, we’ll be running one of the sample pipelines through Spark Submit.`
"Spark doesn't support remote execution" basically means you can't run from Hop Gui (there could be ways to make it work, but those would probably not production-like). We'll see if we can explain this better in the docs.
Can you try your sample again with `spark-submit` like Hans showed and let us know if that helps?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2572962280
since this turned more into a discussion about configuration rather than a bug report, I suggest we close this issue and open a [discussion](https://github.com/apache/hop/discussions/), agreed @Raja10D?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine (hop)
Posted by "Raja10D (via GitHub)" <gi...@apache.org>.
Raja10D commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2569215173
decoders@decoders-Latitude-7480:~/Downloads/apache-hop-client-2.11.0/hop$ spark-submit \
--master local[4] \
--class org.apache.hop.beam.run.MainBeam \
--driver-java-options '-DPROJECT_HOME=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default' \
/home/decoders/Downloads/apache-hop-client-2.11.0/fatjar.jar \
--pipeline=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/beam/pipeline/hop_trans2.hpl \
--config=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/hop-config.json \
--metadata=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/project-config.json \
Spark
25/01/03 18:45:55 WARN Utils: Your hostname, decoders-Latitude-7480 resolves to a loopback address: 127.0.1.1; using 192.125.1.140 instead (on interface enp0s31f6)
25/01/03 18:45:55 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
25/01/03 18:46:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>>>>>> Initializing Hop
log4j:WARN No appenders could be found for logger (org.apache.commons.vfs2.impl.DefaultFileSystemManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Error running Beam pipeline:
Error reading metadata from JSON
Error find metadata class for key null
The metadata plugin for key null could not be found in the plugin registry
org.apache.hop.core.exception.HopException:
Error reading metadata from JSON
Error find metadata class for key null
The metadata plugin for key null could not be found in the plugin registry
at org.apache.hop.core.metadata.SerializableMetadataProvider.<init>(SerializableMetadataProvider.java:143)
at org.apache.hop.beam.run.MainBeam.main(MainBeam.java:101)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hop.core.exception.HopException:
Error find metadata class for key null
The metadata plugin for key null could not be found in the plugin registry
at org.apache.hop.metadata.serializer.BaseMetadataProvider.getMetadataClassForKey(BaseMetadataProvider.java:69)
at org.apache.hop.core.metadata.SerializableMetadataProvider.<init>(SerializableMetadataProvider.java:122)
... 13 more
Caused by: org.apache.hop.core.exception.HopException:
The metadata plugin for key null could not be found in the plugin registry
at org.apache.hop.metadata.serializer.BaseMetadataProvider.getMetadataClassForKey(BaseMetadataProvider.java:61)
... 14 more
**This is the issue I'm Facing
My pc is ubuntu**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org