You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "Raja10D (via GitHub)" <gi...@apache.org> on 2025/01/27 11:22:31 UTC
[I] [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster (hop)
Raja10D opened a new issue, #4835:
URL: https://github.com/apache/hop/issues/4835
### What needs to happen?
I am writing to report an issue we are encountering when running a Hop pipeline on a Spark Standalone cluster.
Our Spark-Submit command works perfectly in a local environment with the following configuration:
**spark-submit \
--master local[4] \
--class org.apache.hop.beam.run.MainBeam \
--driver-java-options '-DPROJECT_HOME=/home/decoders/hop_test_2_8_0_backup/hop/config/projects/default' \
/home/decoders/hop_test_2_8_0_backup/hop/fat-jar.jar \
/home/decoders/hop_test_2_8_0_backup/hop/Creating_5_payoutAmt_source_Dec23.hpl \
/home/decoders/hop_test_2_8_0_backup/hop/metadata.json \
beam_runner_spark**
However, when we replace local[4] with the Spark Standalone cluster URL (spark://10d154:7077), the pipeline fails to run in our real-time production environment. We have verified that all necessary dependencies are included in the fat-jar.jar and reviewed the configuration settings, but the issue persists.
Could you please provide guidance on any additional files or settings that might be required to run the Hop pipeline successfully on a Spark Standalone cluster?
Thank you for your assistance.
### Issue Priority
Priority: 1
### Issue Component
Component: Other, Component: Hop Run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters closed issue #4835: [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster
URL: https://github.com/apache/hop/issues/4835
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #4835:
URL: https://github.com/apache/hop/issues/4835#issuecomment-2615891112
More tips can be found at the documentation of the Hop [Spark Pipeline engine](https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-spark-pipeline-engine.html).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #4835:
URL: https://github.com/apache/hop/issues/4835#issuecomment-2618898818
So no entry for hostname `10d154`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster (hop)
Posted by "Raja10D (via GitHub)" <gi...@apache.org>.
Raja10D commented on issue #4835:
URL: https://github.com/apache/hop/issues/4835#issuecomment-2617946226
I have inserted my spark master web screenshot, and my beam_runner_spark configuration screenshot and spark-submit command
**spark-submit
--master spark://10d154:7077
--class org.apache.hop.beam.run.MainBeam
--driver-java-options '-DPROJECT_HOME=/home/decoders/hop_test_2_8_0_backup/hop/config/projects/default'
/home/decoders/hop_test_2_8_0_backup/hop/fat-jar.jar
/home/decoders/hop_test_2_8_0_backup/hop/Creating_5_payoutAmt_source_Dec23.hpl
/home/decoders/hop_test_2_8_0_backup/hop/metadata.json
beam_runner_spark**
Step by Step I followed the document, can you guide me where am I making mistake.
It will be very helpful.
Thankyou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #4835:
URL: https://github.com/apache/hop/issues/4835#issuecomment-2615815566
Hi @Raja10D, please note that you need to submit this on the master of your production cluster and that you need to specify the master typically as `spark://host:port`, also in the Hop metadata element `beam_runner_spark`.
HTH
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster (hop)
Posted by "Raja10D (via GitHub)" <gi...@apache.org>.
Raja10D commented on issue #4835:
URL: https://github.com/apache/hop/issues/4835#issuecomment-2621265736
> So no entry for hostname `10d154`?
Adding the hostname entry to the /etc/hosts file resolved the problem, and the Hop pipeline is now running smoothly on our Spark Standalone cluster.
**Actual content in my /etc/hosts**
-----------------------------
127.0.0.1 localhost
127.0.1.1 decoders-Latitude-7480
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.125.1.140 master-node worker-node
192.125.1.140 10d154
-----------------------------------
Your guidance was invaluable, and I appreciate your support. Thank you once again for your assistance!
Thanks all..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #4835:
URL: https://github.com/apache/hop/issues/4835#issuecomment-2621274090
You're welcome @Raja10D , I'm glad you managed to run on Spark. Can you resolve this issue and the next?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster (hop)
Posted by "Raja10D (via GitHub)" <gi...@apache.org>.
Raja10D commented on issue #4835:
URL: https://github.com/apache/hop/issues/4835#issuecomment-2615879596
> Hi [@Raja10D](https://github.com/Raja10D), please note that you need to submit this on the master of your production cluster and that you need to specify the master typically as `spark://host:port`, also in the Hop metadata element `beam_runner_spark`. HTH
In this spark://host:port we have to mention our port right if we are running on 8080 (master port ) means **spark://host:8080**
Am I right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster (hop)
Posted by "Raja10D (via GitHub)" <gi...@apache.org>.
Raja10D commented on issue #4835:
URL: https://github.com/apache/hop/issues/4835#issuecomment-2618407409
My /etc/hosts
127.0.0.1 localhost
127.0.1.1 decoders-Latitude-7480
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
# Worker and master node address (same machine)
192.125.1.140 master-node worker-node
My master and worker are in my same machine
My hop - 2.8.0
My beam - 2.50
My Spark - 3.4.4
Java 11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster (hop)
Posted by "Raja10D (via GitHub)" <gi...@apache.org>.
Raja10D commented on issue #4835:
URL: https://github.com/apache/hop/issues/4835#issuecomment-2621304452
> You're welcome [@Raja10D](https://github.com/Raja10D) , I'm glad you managed to run on Spark. Can you resolve this issue and the next?
I'm glad to report that our next step is to run the pipeline in the production environment(in server). We're preparing to ensure that everything runs smoothly there as well.
Are there any specific configurations or considerations we should be aware of before moving forward with the production deployment? Your insights would be invaluable in helping us achieve a seamless transition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #4835:
URL: https://github.com/apache/hop/issues/4835#issuecomment-2615886231
You can see the address at the top of the Spark web console.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on issue #4835:
URL: https://github.com/apache/hop/issues/4835#issuecomment-2618132682
what do the relevant intries the `/etc/hosts` files on your master and other cluster hosts look like?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Issue with Running Hop Pipeline on Spark Standalone Cluster (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #4835:
URL: https://github.com/apache/hop/issues/4835#issuecomment-2621316276
Just look at the transform specific limitations detailed over here: https://hop.apache.org/manual/latest/pipeline/beam/getting-started-with-beam.html#_universal_transforms
In general, if it runs on your local 1 node Spark, it should run on a larger cluster as well.
Good luck!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org