Dataflow java job with java transforms in expansion service

[Sahith Nallapareddy <sa...@spotify.com>]

Hello,

I was wondering if anyone has run a java job with java external transforms
in dataflow? We have had python beam jobs run great with java external
transforms. However, we tried to run a java job with java external
transforms but this seemed to stall on dataflow (this was done a while ago,
have to verify this is still happening). We are working on a system and
having transforms in an expansion service is a component of it. Is this
something that would be supported or is it only going to work cross
language?

Thanks,

Sahith

Re: Dataflow java job with java transforms in expansion service

[Sahith Nallapareddy <sa...@spotify.com>]

Hello,

I ran a job today and it seems like on the latest version of beam this is
working as jobs are actually running now! If I run into any issues i'll
update this thread, but thank you seems like on 2.39 the earlier issues
have been resolved.

Thanks,

Sahith

On Tue, Jun 14, 2022 at 4:14 PM Sahith Nallapareddy <sa...@spotify.com>
wrote:

> Hello,
>
> I will run another one on the latest beam today and let you know what
> happens. The last version I tried this on was I think 2.35. I believe there
> were no errors on the dataflow page, but some issues with getting the
> workers started. I will try on the latest beam and update to see what
> happens!
>
> Thanks,
>
> Sahith
>
> On Tue, Jun 14, 2022 at 12:26 PM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>>
>>
>> On Tue, Jun 14, 2022 at 8:47 AM Sahith Nallapareddy <sa...@spotify.com>
>> wrote:
>>
>>> Hello,
>>>
>>> I was wondering if anyone has run a java job with java external
>>> transforms in dataflow? We have had python beam jobs run great with java
>>> external transforms. However, we tried to run a java job with java external
>>> transforms but this seemed to stall on dataflow (this was done a while ago,
>>> have to verify this is still happening). We are working on a system and
>>> having transforms in an expansion service is a component of it. Is this
>>> something that would be supported or is it only going to work cross
>>> language?
>>>
>>
>> Do you have any specific errors that your job ran into ? In theory this
>> should work (with the latest Beam version) but I don't believe we
>> currently have tests for this.
>> One thing I can think of that could result in jobs getting stuck is
>> forgetting to include "--experiments=use_runner_v2".
>>
>> +Heejong Lee <he...@google.com>
>>
>> Thanks,
>> Cham
>>
>>
>>>
>>> Thanks,
>>>
>>> Sahith
>>>
>>

Re: Dataflow java job with java transforms in expansion service

[Sahith Nallapareddy <sa...@spotify.com>]

Hello,

I will run another one on the latest beam today and let you know what
happens. The last version I tried this on was I think 2.35. I believe there
were no errors on the dataflow page, but some issues with getting the
workers started. I will try on the latest beam and update to see what
happens!

Thanks,

Sahith

On Tue, Jun 14, 2022 at 12:26 PM Chamikara Jayalath <ch...@google.com>
wrote:

>
>
> On Tue, Jun 14, 2022 at 8:47 AM Sahith Nallapareddy <sa...@spotify.com>
> wrote:
>
>> Hello,
>>
>> I was wondering if anyone has run a java job with java external
>> transforms in dataflow? We have had python beam jobs run great with java
>> external transforms. However, we tried to run a java job with java external
>> transforms but this seemed to stall on dataflow (this was done a while ago,
>> have to verify this is still happening). We are working on a system and
>> having transforms in an expansion service is a component of it. Is this
>> something that would be supported or is it only going to work cross
>> language?
>>
>
> Do you have any specific errors that your job ran into ? In theory this
> should work (with the latest Beam version) but I don't believe we
> currently have tests for this.
> One thing I can think of that could result in jobs getting stuck is
> forgetting to include "--experiments=use_runner_v2".
>
> +Heejong Lee <he...@google.com>
>
> Thanks,
> Cham
>
>
>>
>> Thanks,
>>
>> Sahith
>>
>

Re: Dataflow java job with java transforms in expansion service

[Chamikara Jayalath <ch...@google.com>]

On Tue, Jun 14, 2022 at 8:47 AM Sahith Nallapareddy <sa...@spotify.com>
wrote:

> Hello,
>
> I was wondering if anyone has run a java job with java external transforms
> in dataflow? We have had python beam jobs run great with java external
> transforms. However, we tried to run a java job with java external
> transforms but this seemed to stall on dataflow (this was done a while ago,
> have to verify this is still happening). We are working on a system and
> having transforms in an expansion service is a component of it. Is this
> something that would be supported or is it only going to work cross
> language?
>

Do you have any specific errors that your job ran into ? In theory this
should work (with the latest Beam version) but I don't believe we
currently have tests for this.
One thing I can think of that could result in jobs getting stuck is
forgetting to include "--experiments=use_runner_v2".

+Heejong Lee <he...@google.com>

Thanks,
Cham


>
> Thanks,
>
> Sahith
>