You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Richard Primera <ri...@woombatcg.com> on 2018/05/10 16:39:03 UTC

How To Set Environment Variables For Spark Action Script From XML Definition

Greetings,

How can I set an environment variable to be accessible from either a 
.jar or .py script launched via a spark action?

The idea is to set the environment variable with the output of the EL 
function ${wf:id()} from within the XML workflow definition, something 
along these lines:

<jar>script.py</jar>

     <env>OOZIE_WORKFLOW_ID=${wf:id()}</env>

And then have the ability to do wf_id = os.getenv("OOZIE_WORKFLOW_ID") 
from the script without having to pass them as command line arguments. 
The thing about command line arguments is that they don't scale as well 
because they rely on a specific ordering or some custom parsing 
implementation. This can be done easily it seems with a shell action, 
but I've been unable to find a similar straightforward way of doing it 
for a spark action.

Oozie Version: 4.1.0-cdh5.12.1


Re: How To Set Environment Variables For Spark Action Script From XML Definition

Posted by Peter Cseh <ge...@cloudera.com>.
Hi Richard!

I'm happy you've found a workaround for your issue.

Yes, "SPARK_HOME" is set to the current working directory in
SparkActionExecutor
<https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java#L111>

Based on the surrounding code, the It should be merged with user-defined
environment properties though, so the behavior you're experiencing sounds
like a bug to me.
The issue might be that SparkActionExecutor uses "mapred.child.env"
<https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java#L47>
and
not mapred.map.child.env and MapReduce overwrites one with the other
instead of merging them.
gp


On Wed, May 16, 2018 at 11:11 PM, Richard Primera <
richard.primera@woombatcg.com> wrote:

> Greetings,
>
> Thanks for the suggestion. I tried this and noted two things. The first is
> that one has to prepend `oozie.launcher` to the parameter in order for it
> to have an effect over the actual environment of the script. The second, is
> that when I did this the python script exited claiming it couldn't find the
> module pyspark.sql.types, which leads me to believe that
> `mapred.map.child.env` is being used underneath in order to pass some other
> environment variables, and being overwritten by me when I manually set it
> to a particular set of k=v pairs. I don't know if this is the case though,
> just concluding it off the observed behavior.
>
> In the end I managed to get the ${wf:id()} result by appealing to the
> SparkConf object inside the SparkContext provided by Oozie for the spark
> action. I noticed in the stdout log that when the script is run, one of the
> command line parameters given to spark-submit is actually `--conf
> spark.oozie.job.id=${wf:id()}. So in the end, I was lucky and wf_id =
> sc._conf.get("spark.oozie.job.id") did the trick for me from within the
> script. However, I'd still like to find a way of doing it as I originally
> intended, which is by being able to access some environment variable set
> from the XML definition.
>
>
>
> On 05/14/2018 05:58 AM, Peter Cseh wrote:
>
>> Hi!
>>
>> There is no easy and straightforward way of doing this for the Spark
>> action, but you can take advantage of the fact that Oozie 4.1.0 uses
>> MapReduce to launch Spark.
>> Just put "mapred.map.child.env" in the action configuration using the
>> format k1=v1,k2=v2. EL functions should also work here.
>>
>> Gp
>>
>>
>> On Thu, May 10, 2018 at 6:39 PM, Richard Primera <
>> richard.primera@woombatcg.com> wrote:
>>
>> Greetings,
>>>
>>> How can I set an environment variable to be accessible from either a .jar
>>> or .py script launched via a spark action?
>>>
>>> The idea is to set the environment variable with the output of the EL
>>> function ${wf:id()} from within the XML workflow definition, something
>>> along these lines:
>>>
>>> <jar>script.py</jar>
>>>
>>>      <env>OOZIE_WORKFLOW_ID=${wf:id()}</env>
>>>
>>> And then have the ability to do wf_id = os.getenv("OOZIE_WORKFLOW_ID")
>>> from the script without having to pass them as command line arguments.
>>> The
>>> thing about command line arguments is that they don't scale as well
>>> because
>>> they rely on a specific ordering or some custom parsing implementation.
>>> This can be done easily it seems with a shell action, but I've been
>>> unable
>>> to find a similar straightforward way of doing it for a spark action.
>>>
>>> Oozie Version: 4.1.0-cdh5.12.1
>>>
>>>
>>>
>>
>


-- 
*Peter Cseh *| Software Engineer
cloudera.com <https://www.cloudera.com>

[image: Cloudera] <https://www.cloudera.com/>

[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------

Re: How To Set Environment Variables For Spark Action Script From XML Definition

Posted by Richard Primera <ri...@woombatcg.com>.
Greetings,

Thanks for the suggestion. I tried this and noted two things. The first 
is that one has to prepend `oozie.launcher` to the parameter in order 
for it to have an effect over the actual environment of the script. The 
second, is that when I did this the python script exited claiming it 
couldn't find the module pyspark.sql.types, which leads me to believe 
that `mapred.map.child.env` is being used underneath in order to pass 
some other environment variables, and being overwritten by me when I 
manually set it to a particular set of k=v pairs. I don't know if this 
is the case though, just concluding it off the observed behavior.

In the end I managed to get the ${wf:id()} result by appealing to the 
SparkConf object inside the SparkContext provided by Oozie for the spark 
action. I noticed in the stdout log that when the script is run, one of 
the command line parameters given to spark-submit is actually `--conf 
spark.oozie.job.id=${wf:id()}. So in the end, I was lucky and wf_id = 
sc._conf.get("spark.oozie.job.id") did the trick for me from within the 
script. However, I'd still like to find a way of doing it as I 
originally intended, which is by being able to access some environment 
variable set from the XML definition.


On 05/14/2018 05:58 AM, Peter Cseh wrote:
> Hi!
>
> There is no easy and straightforward way of doing this for the Spark
> action, but you can take advantage of the fact that Oozie 4.1.0 uses
> MapReduce to launch Spark.
> Just put "mapred.map.child.env" in the action configuration using the
> format k1=v1,k2=v2. EL functions should also work here.
>
> Gp
>
>
> On Thu, May 10, 2018 at 6:39 PM, Richard Primera <
> richard.primera@woombatcg.com> wrote:
>
>> Greetings,
>>
>> How can I set an environment variable to be accessible from either a .jar
>> or .py script launched via a spark action?
>>
>> The idea is to set the environment variable with the output of the EL
>> function ${wf:id()} from within the XML workflow definition, something
>> along these lines:
>>
>> <jar>script.py</jar>
>>
>>      <env>OOZIE_WORKFLOW_ID=${wf:id()}</env>
>>
>> And then have the ability to do wf_id = os.getenv("OOZIE_WORKFLOW_ID")
>> from the script without having to pass them as command line arguments. The
>> thing about command line arguments is that they don't scale as well because
>> they rely on a specific ordering or some custom parsing implementation.
>> This can be done easily it seems with a shell action, but I've been unable
>> to find a similar straightforward way of doing it for a spark action.
>>
>> Oozie Version: 4.1.0-cdh5.12.1
>>
>>
>


Re: How To Set Environment Variables For Spark Action Script From XML Definition

Posted by Peter Cseh <ge...@cloudera.com>.
Hi!

There is no easy and straightforward way of doing this for the Spark
action, but you can take advantage of the fact that Oozie 4.1.0 uses
MapReduce to launch Spark.
Just put "mapred.map.child.env" in the action configuration using the
format k1=v1,k2=v2. EL functions should also work here.

Gp


On Thu, May 10, 2018 at 6:39 PM, Richard Primera <
richard.primera@woombatcg.com> wrote:

> Greetings,
>
> How can I set an environment variable to be accessible from either a .jar
> or .py script launched via a spark action?
>
> The idea is to set the environment variable with the output of the EL
> function ${wf:id()} from within the XML workflow definition, something
> along these lines:
>
> <jar>script.py</jar>
>
>     <env>OOZIE_WORKFLOW_ID=${wf:id()}</env>
>
> And then have the ability to do wf_id = os.getenv("OOZIE_WORKFLOW_ID")
> from the script without having to pass them as command line arguments. The
> thing about command line arguments is that they don't scale as well because
> they rely on a specific ordering or some custom parsing implementation.
> This can be done easily it seems with a shell action, but I've been unable
> to find a similar straightforward way of doing it for a spark action.
>
> Oozie Version: 4.1.0-cdh5.12.1
>
>


-- 
*Peter Cseh *| Software Engineer
cloudera.com <https://www.cloudera.com>

[image: Cloudera] <https://www.cloudera.com/>

[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------