Passing "conf" arguments using a portable runner in Java (spark job runner)

[Jon Molle via user <us...@beam.apache.org>]

Hi,

I'm having trouble trying to figure out where to pass "conf" spark-submit
arguments to a spark job service. I don't particularly care at this point
whether the job service uses the same set of args for all jobs passed to
the service, I'm just having trouble finding where I can send these args
where they will get picked up by the service.

For reference, I'm using the portable pipeline options (IE: the portable
runner, job endpoint, and DOCKER environment type).

Has anyone tried this, specifically in Java? I'm actually writing in
Kotlin, but all the examples of the portable runner are in Python, which is
significantly different from the Java design.

Thanks!

Re: Passing "conf" arguments using a portable runner in Java (spark job runner)

[Moritz Mack <mm...@talend.com>]

Hi Jon,

sorry for the late replay.

A while ago I was struggling with as well. Unfortunately, there’s no direct way to do this per pipeline.
However, you can set default arguments by passing them to the job service container using the environment variable _JAVA_OPTIONS.

I hope this still helps!
Cheers,
Moritz



On 30.06.23, 20:29, "Jon Molle via user" <us...@beam.apache.org> wrote:

Hi, I'm having trouble trying to figure out where to pass "conf" spark-submit arguments to a spark job service. I don't particularly care at this point whether the job service uses the same set of args for all jobs passed

Hi,

I'm having trouble trying to figure out where to pass "conf" spark-submit arguments to a spark job service. I don't particularly care at this point whether the job service uses the same set of args for all jobs passed to the service, I'm just having trouble finding where I can send these args where they will get picked up by the service.

For reference, I'm using the portable pipeline options (IE: the portable runner, job endpoint, and DOCKER environment type).

Has anyone tried this, specifically in Java? I'm actually writing in Kotlin, but all the examples of the portable runner are in Python, which is significantly different from the Java design.

Thanks!

As a recipient of an email from the Talend Group, your personal data will be processed by our systems. Please see our Privacy Notice <https://www.talend.com/privacy-policy/> for more information about our collection and use of your personal information, our security practices, and your data protection rights, including any rights you may have to object to automated-decision making or profiling we use to analyze support or marketing related communications. To manage or discontinue promotional communications, use the communication preferences portal<https://info.talend.com/emailpreferencesen.html>. To exercise your data protection rights, use the privacy request form<https://talend.my.onetrust.com/webform/ef906c5a-de41-4ea0-ba73-96c079cdd15a/b191c71d-f3cb-4a42-9815-0c3ca021704cl>. Contact us here <https://www.talend.com/contact/> or by mail to either of our co-headquarters: Talend, Inc.: 400 South El Camino Real, Ste 1400, San Mateo, CA 94402; Talend SAS: 5/7 rue Salomon De Rothschild, 92150 Suresnes, France