You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Sahith Nallapareddy via dev <de...@beam.apache.org> on 2023/04/24 17:56:57 UTC
Testing a pipeline with external transform
Hello,
I was wondering the best way to test a pipeline with external transform. We
were trying to use the DirectRunner but it seemed to hang on the external
transform step. I was wondering if someone could point me to somewhere in
the beam code where this is done or give some pointers on how to test a
pipeline with an External step in it? I tried searching a bit but was
unsuccessful, but maybe I was looking in the wrong place.
Thanks,
Sahith
Re: Testing a pipeline with external transform
Posted by Sahith Nallapareddy via dev <de...@beam.apache.org>.
Sounds good, let me try that thank you again!
Thanks,
Sahith
On Mon, Apr 24, 2023 at 2:11 PM Chamikara Jayalath <ch...@google.com>
wrote:
>
>
> On Mon, Apr 24, 2023 at 11:06 AM Sahith Nallapareddy <sa...@spotify.com>
> wrote:
>
>> Hello,
>>
>> Ah I missed that thank you! I am assuming for java jobs with java
>> external transform, we run a Java Job Service?
>>
>
> No, the job service is tied to the runner not to the pipeline SDK. The
> Java quickstart guide gives instructions for starting up a job service for
> Python portable runner for running Java jobs.
>
> - Cham
>
>
>
>>
>> Thanks,
>>
>> Sahith
>>
>> On Mon, Apr 24, 2023 at 2:01 PM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> Have you looked into quickstart guides (Java [1], Python [2]) which also
>>> gives instructions for running using DirectRunner ?
>>>
>>> Thanks,
>>> Cham
>>>
>>> [1]
>>> https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/
>>> [2]
>>> https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/
>>>
>>> On Mon, Apr 24, 2023 at 10:57 AM Sahith Nallapareddy via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> Hello,
>>>>
>>>> I was wondering the best way to test a pipeline with external
>>>> transform. We were trying to use the DirectRunner but it seemed to hang on
>>>> the external transform step. I was wondering if someone could point me to
>>>> somewhere in the beam code where this is done or give some pointers on how
>>>> to test a pipeline with an External step in it? I tried searching a bit but
>>>> was unsuccessful, but maybe I was looking in the wrong place.
>>>>
>>>> Thanks,
>>>>
>>>> Sahith
>>>>
>>>
Re: Testing a pipeline with external transform
Posted by Chamikara Jayalath via dev <de...@beam.apache.org>.
On Mon, Apr 24, 2023 at 11:06 AM Sahith Nallapareddy <sa...@spotify.com>
wrote:
> Hello,
>
> Ah I missed that thank you! I am assuming for java jobs with java external
> transform, we run a Java Job Service?
>
No, the job service is tied to the runner not to the pipeline SDK. The Java
quickstart guide gives instructions for starting up a job service for
Python portable runner for running Java jobs.
- Cham
>
> Thanks,
>
> Sahith
>
> On Mon, Apr 24, 2023 at 2:01 PM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> Have you looked into quickstart guides (Java [1], Python [2]) which also
>> gives instructions for running using DirectRunner ?
>>
>> Thanks,
>> Cham
>>
>> [1]
>> https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/
>> [2]
>> https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/
>>
>> On Mon, Apr 24, 2023 at 10:57 AM Sahith Nallapareddy via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hello,
>>>
>>> I was wondering the best way to test a pipeline with external transform.
>>> We were trying to use the DirectRunner but it seemed to hang on the
>>> external transform step. I was wondering if someone could point me to
>>> somewhere in the beam code where this is done or give some pointers on how
>>> to test a pipeline with an External step in it? I tried searching a bit but
>>> was unsuccessful, but maybe I was looking in the wrong place.
>>>
>>> Thanks,
>>>
>>> Sahith
>>>
>>
Re: Testing a pipeline with external transform
Posted by Sahith Nallapareddy via dev <de...@beam.apache.org>.
Hello,
Ah I missed that thank you! I am assuming for java jobs with java external
transform, we run a Java Job Service?
Thanks,
Sahith
On Mon, Apr 24, 2023 at 2:01 PM Chamikara Jayalath <ch...@google.com>
wrote:
> Have you looked into quickstart guides (Java [1], Python [2]) which also
> gives instructions for running using DirectRunner ?
>
> Thanks,
> Cham
>
> [1]
> https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/
> [2]
> https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/
>
> On Mon, Apr 24, 2023 at 10:57 AM Sahith Nallapareddy via dev <
> dev@beam.apache.org> wrote:
>
>> Hello,
>>
>> I was wondering the best way to test a pipeline with external transform.
>> We were trying to use the DirectRunner but it seemed to hang on the
>> external transform step. I was wondering if someone could point me to
>> somewhere in the beam code where this is done or give some pointers on how
>> to test a pipeline with an External step in it? I tried searching a bit but
>> was unsuccessful, but maybe I was looking in the wrong place.
>>
>> Thanks,
>>
>> Sahith
>>
>
Re: Testing a pipeline with external transform
Posted by Chamikara Jayalath via dev <de...@beam.apache.org>.
Have you looked into quickstart guides (Java [1], Python [2]) which also
gives instructions for running using DirectRunner ?
Thanks,
Cham
[1]
https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/
[2]
https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/
On Mon, Apr 24, 2023 at 10:57 AM Sahith Nallapareddy via dev <
dev@beam.apache.org> wrote:
> Hello,
>
> I was wondering the best way to test a pipeline with external transform.
> We were trying to use the DirectRunner but it seemed to hang on the
> external transform step. I was wondering if someone could point me to
> somewhere in the beam code where this is done or give some pointers on how
> to test a pipeline with an External step in it? I tried searching a bit but
> was unsuccessful, but maybe I was looking in the wrong place.
>
> Thanks,
>
> Sahith
>