You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Ana Markovic <am...@york.ac.uk> on 2021/08/26 08:36:04 UTC
[TROUBLESHOOT] Multi-language pipeline errors
To whom it may concern,
Recently I started experimenting with Apache Beam's cross-language features
and I run into a couple of problems. I'll explain them here, and share the
code with running instructions in a GitHub repository. I truly hope you
will be able to help me fix these issues!
1. Python pipeline with a Java-based external transform
1. When Python sends a PCollection<String> and gets the
PCollection<String> it works as expected, the code for that is
posted in this
commit
<https://github.com/ana-markovic/multi-language-pipeline-beam/tree/ffd995049bc3a26981041c37f2bcee72f57d3a74>
2. When Python sends a PCollection<String> and expects a
PCollection<KV<String, Long>>, the pipeline compiles but I get a
RuntimeError you can see in the image attached. Here is the link to
that commit
<https://github.com/ana-markovic/multi-language-pipeline-beam/tree/py-java-wordcount>.
Do you maybe know what causes this error in communication and
how to solve
it?
2. Java pipeline with an external transform written in Python following this
page of the documentation
<https://beam.apache.org/documentation/patterns/cross-language/>, runs
indefinitely with no error or result... Like the execution is stuck. The
code can be found here
<https://github.com/ana-markovic/multi-language-pipeline-beam/tree/java-py-wordcount>
.
Thanks in advance for your assistance and feel free to let me know if you
need anything else from my side.
Best,
Ana
Re: [TROUBLESHOOT] Multi-language pipeline errors
Posted by Chamikara Jayalath <ch...@google.com>.
On Fri, Aug 27, 2021 at 1:33 AM Ana Markovic <am...@york.ac.uk> wrote:
> Thank you!
>
> And do you, by any chance, know why the Java-Py pipeline doesn't execute?
>
+Heejong Lee <he...@google.com> might be able to help here. Note that
using Python transforms from Java is still in development and APIs and
implementation is not complete yet.
Thanks,
Cham
>
>
>> 1. Java pipeline with an external transform written in Python
>> following this page of the documentation
>> <https://beam.apache.org/documentation/patterns/cross-language/>,
>> runs indefinitely with no error or result... Like the execution is stuck.
>> The code can be found here
>> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/java-py-wordcount>
>> .
>>
>>
>
>
> Thanks,
> Ana
>
>
> On Thu, 26 Aug 2021 at 17:40, Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> Usually this happens when certain Transform translation classes are not
>> registered due to not being in the classpath when starting up the expansion
>> service. I created https://issues.apache.org/jira/browse/BEAM-12807 to
>> improve the user experience here.
>>
>> Thanks,
>> Cham
>>
>> On Thu, Aug 26, 2021 at 2:54 AM Ana Markovic <am...@york.ac.uk> wrote:
>>
>>> Hi Chamikara,
>>>
>>> Thanks for the suggestion. I added it, however I still get the same
>>> error (*"java.lang.IllegalArgumentException: Cannot process transform
>>> with no spec"*) when trying to send a PCollection of Strings and
>>> receive KV pairs back.
>>>
>>> Please share if anything else comes to mind!
>>>
>>> Best,
>>> Ana
>>>
>>>
>>> On Thu, 26 Aug 2021 at 10:51, Chamikara Jayalath <ch...@google.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Aug 26, 2021 at 1:36 AM Ana Markovic <am...@york.ac.uk> wrote:
>>>>
>>>>> To whom it may concern,
>>>>>
>>>>> Recently I started experimenting with Apache Beam's cross-language
>>>>> features and I run into a couple of problems. I'll explain them here, and
>>>>> share the code with running instructions in a GitHub repository. I truly
>>>>> hope you will be able to help me fix these issues!
>>>>>
>>>>>
>>>>> 1. Python pipeline with a Java-based external transform
>>>>> 1. When Python sends a PCollection<String> and gets the
>>>>> PCollection<String> it works as expected, the code for that is posted in this
>>>>> commit
>>>>> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/ffd995049bc3a26981041c37f2bcee72f57d3a74>
>>>>> 2. When Python sends a PCollection<String> and expects a
>>>>> PCollection<KV<String, Long>>, the pipeline compiles but I get a
>>>>> RuntimeError you can see in the image attached. Here is the link
>>>>> to that commit
>>>>> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/py-java-wordcount>.
>>>>> Do you maybe know what causes this error in communication and how to solve
>>>>> it?
>>>>>
>>>>>
>>>> Can you try adding the following dependency to your pom.xml file ?
>>>>
>>>> <dependency>
>>>> <groupId>org.apache.beam</groupId>
>>>> <artifactId>beam-runners-core-construction-java</artifactId>
>>>> <version>${beam.version}</version>
>>>> </dependency>
>>>>
>>>>
>>>>>
>>>>> 1. Java pipeline with an external transform written in Python
>>>>> following this page of the documentation
>>>>> <https://beam.apache.org/documentation/patterns/cross-language/>,
>>>>> runs indefinitely with no error or result... Like the execution is stuck.
>>>>> The code can be found here
>>>>> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/java-py-wordcount>
>>>>> .
>>>>>
>>>>>
>>>>> Thanks in advance for your assistance and feel free to let me know if
>>>>> you need anything else from my side.
>>>>>
>>>>> Best,
>>>>> Ana
>>>>>
>>>>
Re: [TROUBLESHOOT] Multi-language pipeline errors
Posted by Ana Markovic <am...@york.ac.uk>.
Thank you!
And do you, by any chance, know why the Java-Py pipeline doesn't execute?
> 1. Java pipeline with an external transform written in Python following
> this page of the documentation
> <https://beam.apache.org/documentation/patterns/cross-language/>, runs
> indefinitely with no error or result... Like the execution is stuck. The
> code can be found here
> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/java-py-wordcount>
> .
>
>
Thanks,
Ana
On Thu, 26 Aug 2021 at 17:40, Chamikara Jayalath <ch...@google.com>
wrote:
> Usually this happens when certain Transform translation classes are not
> registered due to not being in the classpath when starting up the expansion
> service. I created https://issues.apache.org/jira/browse/BEAM-12807 to
> improve the user experience here.
>
> Thanks,
> Cham
>
> On Thu, Aug 26, 2021 at 2:54 AM Ana Markovic <am...@york.ac.uk> wrote:
>
>> Hi Chamikara,
>>
>> Thanks for the suggestion. I added it, however I still get the same error
>> (*"java.lang.IllegalArgumentException: Cannot process transform with no
>> spec"*) when trying to send a PCollection of Strings and receive KV
>> pairs back.
>>
>> Please share if anything else comes to mind!
>>
>> Best,
>> Ana
>>
>>
>> On Thu, 26 Aug 2021 at 10:51, Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>>
>>>
>>> On Thu, Aug 26, 2021 at 1:36 AM Ana Markovic <am...@york.ac.uk> wrote:
>>>
>>>> To whom it may concern,
>>>>
>>>> Recently I started experimenting with Apache Beam's cross-language
>>>> features and I run into a couple of problems. I'll explain them here, and
>>>> share the code with running instructions in a GitHub repository. I truly
>>>> hope you will be able to help me fix these issues!
>>>>
>>>>
>>>> 1. Python pipeline with a Java-based external transform
>>>> 1. When Python sends a PCollection<String> and gets the
>>>> PCollection<String> it works as expected, the code for that is posted in this
>>>> commit
>>>> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/ffd995049bc3a26981041c37f2bcee72f57d3a74>
>>>> 2. When Python sends a PCollection<String> and expects a
>>>> PCollection<KV<String, Long>>, the pipeline compiles but I get a
>>>> RuntimeError you can see in the image attached. Here is the link
>>>> to that commit
>>>> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/py-java-wordcount>.
>>>> Do you maybe know what causes this error in communication and how to solve
>>>> it?
>>>>
>>>>
>>> Can you try adding the following dependency to your pom.xml file ?
>>>
>>> <dependency>
>>> <groupId>org.apache.beam</groupId>
>>> <artifactId>beam-runners-core-construction-java</artifactId>
>>> <version>${beam.version}</version>
>>> </dependency>
>>>
>>>
>>>>
>>>> 1. Java pipeline with an external transform written in Python
>>>> following this page of the documentation
>>>> <https://beam.apache.org/documentation/patterns/cross-language/>,
>>>> runs indefinitely with no error or result... Like the execution is stuck.
>>>> The code can be found here
>>>> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/java-py-wordcount>
>>>> .
>>>>
>>>>
>>>> Thanks in advance for your assistance and feel free to let me know if
>>>> you need anything else from my side.
>>>>
>>>> Best,
>>>> Ana
>>>>
>>>
Re: [TROUBLESHOOT] Multi-language pipeline errors
Posted by Chamikara Jayalath <ch...@google.com>.
Usually this happens when certain Transform translation classes are not
registered due to not being in the classpath when starting up the expansion
service. I created https://issues.apache.org/jira/browse/BEAM-12807 to
improve the user experience here.
Thanks,
Cham
On Thu, Aug 26, 2021 at 2:54 AM Ana Markovic <am...@york.ac.uk> wrote:
> Hi Chamikara,
>
> Thanks for the suggestion. I added it, however I still get the same error (*"java.lang.IllegalArgumentException:
> Cannot process transform with no spec"*) when trying to send a
> PCollection of Strings and receive KV pairs back.
>
> Please share if anything else comes to mind!
>
> Best,
> Ana
>
>
> On Thu, 26 Aug 2021 at 10:51, Chamikara Jayalath <ch...@google.com>
> wrote:
>
>>
>>
>> On Thu, Aug 26, 2021 at 1:36 AM Ana Markovic <am...@york.ac.uk> wrote:
>>
>>> To whom it may concern,
>>>
>>> Recently I started experimenting with Apache Beam's cross-language
>>> features and I run into a couple of problems. I'll explain them here, and
>>> share the code with running instructions in a GitHub repository. I truly
>>> hope you will be able to help me fix these issues!
>>>
>>>
>>> 1. Python pipeline with a Java-based external transform
>>> 1. When Python sends a PCollection<String> and gets the
>>> PCollection<String> it works as expected, the code for that is posted in this
>>> commit
>>> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/ffd995049bc3a26981041c37f2bcee72f57d3a74>
>>> 2. When Python sends a PCollection<String> and expects a
>>> PCollection<KV<String, Long>>, the pipeline compiles but I get a
>>> RuntimeError you can see in the image attached. Here is the link
>>> to that commit
>>> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/py-java-wordcount>.
>>> Do you maybe know what causes this error in communication and how to solve
>>> it?
>>>
>>>
>> Can you try adding the following dependency to your pom.xml file ?
>>
>> <dependency>
>> <groupId>org.apache.beam</groupId>
>> <artifactId>beam-runners-core-construction-java</artifactId>
>> <version>${beam.version}</version>
>> </dependency>
>>
>>
>>>
>>> 1. Java pipeline with an external transform written in Python
>>> following this page of the documentation
>>> <https://beam.apache.org/documentation/patterns/cross-language/>,
>>> runs indefinitely with no error or result... Like the execution is stuck.
>>> The code can be found here
>>> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/java-py-wordcount>
>>> .
>>>
>>>
>>> Thanks in advance for your assistance and feel free to let me know if
>>> you need anything else from my side.
>>>
>>> Best,
>>> Ana
>>>
>>
Re: [TROUBLESHOOT] Multi-language pipeline errors
Posted by Ana Markovic <am...@york.ac.uk>.
Hi Chamikara,
Thanks for the suggestion. I added it, however I still get the same
error (*"java.lang.IllegalArgumentException:
Cannot process transform with no spec"*) when trying to send a PCollection
of Strings and receive KV pairs back.
Please share if anything else comes to mind!
Best,
Ana
On Thu, 26 Aug 2021 at 10:51, Chamikara Jayalath <ch...@google.com>
wrote:
>
>
> On Thu, Aug 26, 2021 at 1:36 AM Ana Markovic <am...@york.ac.uk> wrote:
>
>> To whom it may concern,
>>
>> Recently I started experimenting with Apache Beam's cross-language
>> features and I run into a couple of problems. I'll explain them here, and
>> share the code with running instructions in a GitHub repository. I truly
>> hope you will be able to help me fix these issues!
>>
>>
>> 1. Python pipeline with a Java-based external transform
>> 1. When Python sends a PCollection<String> and gets the
>> PCollection<String> it works as expected, the code for that is posted in this
>> commit
>> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/ffd995049bc3a26981041c37f2bcee72f57d3a74>
>> 2. When Python sends a PCollection<String> and expects a
>> PCollection<KV<String, Long>>, the pipeline compiles but I get a
>> RuntimeError you can see in the image attached. Here is the link
>> to that commit
>> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/py-java-wordcount>.
>> Do you maybe know what causes this error in communication and how to solve
>> it?
>>
>>
> Can you try adding the following dependency to your pom.xml file ?
>
> <dependency>
> <groupId>org.apache.beam</groupId>
> <artifactId>beam-runners-core-construction-java</artifactId>
> <version>${beam.version}</version>
> </dependency>
>
>
>>
>> 1. Java pipeline with an external transform written in Python
>> following this page of the documentation
>> <https://beam.apache.org/documentation/patterns/cross-language/>,
>> runs indefinitely with no error or result... Like the execution is stuck.
>> The code can be found here
>> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/java-py-wordcount>
>> .
>>
>>
>> Thanks in advance for your assistance and feel free to let me know if you
>> need anything else from my side.
>>
>> Best,
>> Ana
>>
>
Re: [TROUBLESHOOT] Multi-language pipeline errors
Posted by Chamikara Jayalath <ch...@google.com>.
On Thu, Aug 26, 2021 at 1:36 AM Ana Markovic <am...@york.ac.uk> wrote:
> To whom it may concern,
>
> Recently I started experimenting with Apache Beam's cross-language
> features and I run into a couple of problems. I'll explain them here, and
> share the code with running instructions in a GitHub repository. I truly
> hope you will be able to help me fix these issues!
>
>
> 1. Python pipeline with a Java-based external transform
> 1. When Python sends a PCollection<String> and gets the
> PCollection<String> it works as expected, the code for that is posted in this
> commit
> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/ffd995049bc3a26981041c37f2bcee72f57d3a74>
> 2. When Python sends a PCollection<String> and expects a
> PCollection<KV<String, Long>>, the pipeline compiles but I get a
> RuntimeError you can see in the image attached. Here is the link to
> that commit
> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/py-java-wordcount>.
> Do you maybe know what causes this error in communication and how to solve
> it?
>
>
Can you try adding the following dependency to your pom.xml file ?
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-core-construction-java</artifactId>
<version>${beam.version}</version>
</dependency>
>
> 1. Java pipeline with an external transform written in Python
> following this page of the documentation
> <https://beam.apache.org/documentation/patterns/cross-language/>, runs
> indefinitely with no error or result... Like the execution is stuck. The
> code can be found here
> <https://github.com/ana-markovic/multi-language-pipeline-beam/tree/java-py-wordcount>
> .
>
>
> Thanks in advance for your assistance and feel free to let me know if you
> need anything else from my side.
>
> Best,
> Ana
>