You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hop.apache.org by Raja Ganesh <ra...@gmail.com> on 2025/01/02 10:14:59 UTC
Issue with Running Hop Pipeline on Beam Spark Pipeline Engine
Dear Apache Hop Team,
I am reaching out to seek your assistance with an issue I am encountering
while running a simple Hop pipeline using the Beam Spark pipeline engine.
The pipeline executes without any issues when using the default local
runner, but it fails when I switch to the Beam Spark engine.
Below are the details of my setup:
- *Apache Beam Version:* 2.61.0
- *Apache Spark Version:* 3.5.1
- *Apache Hop Version:* apache-hop-client-2.11.0
The error logs are as follows:
yaml
Copy code
2025/01/02 15:30:53 - Hop - Pipeline opened. 2025/01/02 15:30:53 -
Hop - Launching pipeline [hop_trans2]... 2025/01/02 15:30:53 - Hop -
Started the pipeline execution. java.lang.NoClassDefFoundError:
org/apache/spark/metrics/source/Source
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
... 2025/01/02 15:30:53 - Hop - ERROR: hop_trans2: preparing
pipeline execution failed 2025/01/02 15:30:53 - Hop - ERROR:
org.apache.hop.core.exception.HopException: 2025/01/02 15:30:53 - Hop
- Error preparing remote pipeline 2025/01/02 15:30:53 - Hop - Error
converting Hop pipeline to Beam 2025/01/02 15:30:53 - Hop -
2025/01/02 15:30:53 - Hop - Caused by: java.lang.NoClassDefFoundError:
org/apache/spark/metrics/source/Source 2025/01/02 15:30:53 - Hop -
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
...
It appears that the issue is related to the
org.apache.spark.metrics.source.Source class not being found.
I have also attached the following:
1. *Pipeline Configuration Screenshot*
2. *Pipeline Screenshot*
3. *Complete Error Log*
Could you please provide guidance on how to resolve this issue? Let me know
if there are any additional steps or configurations required to
successfully run the pipeline on the Beam Spark pipeline engine.
Thank you for your support.
Best regards,
Rajaganesh S
Re: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine
Posted by "Matt Casters (i-Bridge)" <ma...@ibridge.be>.
Hi Rajaganesh,
Class-not-found exceptions are usually seen if you didn't build a fat jar
or if there's a mismatch between your Spark version and the one included in
Hop.
It's best to follow the Getting Started With Beam page in the
documentation.
Good luck!
Matt
Op do 2 jan 2025 om 12:12 schreef Raja Ganesh <
rajaganesh18beemepcoeng@gmail.com>:
> Dear Apache Hop Team,
>
> I am reaching out to seek your assistance with an issue I am encountering
> while running a simple Hop pipeline using the Beam Spark pipeline engine.
> The pipeline executes without any issues when using the default local
> runner, but it fails when I switch to the Beam Spark engine.
>
> Below are the details of my setup:
>
> - *Apache Beam Version:* 2.61.0
> - *Apache Spark Version:* 3.5.1
> - *Apache Hop Version:* apache-hop-client-2.11.0
>
> The error logs are as follows:
>
> yaml
> Copy code
> 2025/01/02 15:30:53 - Hop - Pipeline opened. 2025/01/02 15:30:53 - Hop - Launching pipeline [hop_trans2]... 2025/01/02 15:30:53 - Hop - Started the pipeline execution. java.lang.NoClassDefFoundError: org/apache/spark/metrics/source/Source
> at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
> ... 2025/01/02 15:30:53 - Hop - ERROR: hop_trans2: preparing pipeline execution failed 2025/01/02 15:30:53 - Hop - ERROR: org.apache.hop.core.exception.HopException: 2025/01/02 15:30:53 - Hop - Error preparing remote pipeline 2025/01/02 15:30:53 - Hop - Error converting Hop pipeline to Beam 2025/01/02 15:30:53 - Hop - 2025/01/02 15:30:53 - Hop - Caused by: java.lang.NoClassDefFoundError: org/apache/spark/metrics/source/Source 2025/01/02 15:30:53 - Hop - at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
> ...
>
> It appears that the issue is related to the
> org.apache.spark.metrics.source.Source class not being found.
>
> I have also attached the following:
>
> 1. *Pipeline Configuration Screenshot*
> 2. *Pipeline Screenshot*
> 3. *Complete Error Log*
>
> Could you please provide guidance on how to resolve this issue? Let me
> know if there are any additional steps or configurations required to
> successfully run the pipeline on the Beam Spark pipeline engine.
>
> Thank you for your support.
>
> Best regards,
> Rajaganesh S
>