Questions about the Spark Action

[Lars Francke <la...@gmail.com>]

Hi,

I'm wondering what the <configuration> element can be used for in the Spark
Action. It seems as if it is not used in any way and properties are not
forwarded to Spark.

I also wonder why the Spark action does not write the oozie.action.conf.xml
file and/or property. What is puzzling about this for me is that looking at
the code I don't see why this is not happening as it inherits the relevant
parts from the Java action. I must be missing something obvious.

Any input is appreciated, thank you.

Lars

Re: FW: Questions about the Spark Action

[Lars Francke <la...@gmail.com>]

Hi Pavan,

thanks for your mail.

On Wed, Jul 15, 2015 at 8:22 AM, pavan kumar Kolamuri <
pavan.kolamuri@gmail.com> wrote:

> Hi Lars
>
> 1) <configuration> element can be used to pass properties to oozie
> launcher job of spark.
> ex: If some one wants to update the priority of launcher job. or in spark
> user class precedence was set that also can be overrided by properties in
> configuration
>

Okay, that can be. The documentation states "The configuration element, if
present, contains configuration properties that are passed to the Spark
job." and that does not seem to be the case. The driver is not aware of the
properties at all.

2) I have run spark jobs , in fact action.xml file is actually getting
> written.  It was written in
> /user/oozie/oozie-oozi/0000000-150422124453092-oozie-oozi-W/spark-node--spark/action.xml
> in this path in my case
>

That is not what I mean. The Java action writes a file to the local
filesystem (as far as I know) so that the Java code can read the properties
in: <
https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a3.2.7_Java_Action>
But that is not happening here.

See these two pastes:
<http://pastebin.com/GfDyFvnh>
<http://pastebin.com/UGwm5dAM>



> Please feel free to correct me if i am missing something
>
>
> On Wed, Jul 15, 2015 at 10:02 AM, Shwetha Shivalingamurthy <
> sshivalingamurthy@hortonworks.com> wrote:
>
>> Can you reply?
>>
>>
>> On 14/07/15 7:27 pm, "Lars Francke" <la...@gmail.com> wrote:
>>
>> >Hi,
>> >
>> >I'm wondering what the <configuration> element can be used for in the
>> >Spark
>> >Action. It seems as if it is not used in any way and properties are not
>> >forwarded to Spark.
>> >
>> >I also wonder why the Spark action does not write the
>> >oozie.action.conf.xml
>> >file and/or property. What is puzzling about this for me is that looking
>> >at
>> >the code I don't see why this is not happening as it inherits the
>> relevant
>> >parts from the Java action. I must be missing something obvious.
>> >
>> >Any input is appreciated, thank you.
>> >
>> >Lars
>>
>>
>
>
> --
> Regards
> Pavan Kumar Kolamuri
>
>

Re: FW: Questions about the Spark Action

[pavan kumar Kolamuri <pa...@gmail.com>]

Hi Lars

1) <configuration> element can be used to pass properties to oozie launcher
job of spark.
ex: If some one wants to update the priority of launcher job. or in spark
user class precedence was set that also can be overrided by properties in
configuration

2) I have run spark jobs , in fact action.xml file is actually getting
written.  It was written in
/user/oozie/oozie-oozi/0000000-150422124453092-oozie-oozi-W/spark-node--spark/action.xml
in this path in my case

Please feel free to correct me if i am missing something


On Wed, Jul 15, 2015 at 10:02 AM, Shwetha Shivalingamurthy <
sshivalingamurthy@hortonworks.com> wrote:

> Can you reply?
>
> On 14/07/15 7:27 pm, "Lars Francke" <la...@gmail.com> wrote:
>
> >Hi,
> >
> >I'm wondering what the <configuration> element can be used for in the
> >Spark
> >Action. It seems as if it is not used in any way and properties are not
> >forwarded to Spark.
> >
> >I also wonder why the Spark action does not write the
> >oozie.action.conf.xml
> >file and/or property. What is puzzling about this for me is that looking
> >at
> >the code I don't see why this is not happening as it inherits the relevant
> >parts from the Java action. I must be missing something obvious.
> >
> >Any input is appreciated, thank you.
> >
> >Lars
>
>


-- 
Regards
Pavan Kumar Kolamuri