ClassLoader resources on executor

[Charles Allen <ch...@metamarkets.com>]

Is there a way to pass configuration file resources to be resolvable
through the classloader?

For example, if I'm using a library (non-spark) that can use a
some-lib.properties file in the classpath/classLoader, can I pass that file
so that when it tries to get the resource from the classloader it is able
to find it?

One potential solution is to take the files and package them as resources
in a jar, and include the jar as part of the spark job, but that feels like
a hack instead of an actual solution.

Is there any support or planned support for such a thing?

https://github.com/apache/spark/pull/9118 seems to tackle a similar problem
in a hard-coded kind of way.

Thank you,
Charles Allen

Re: ClassLoader resources on executor

[Marcelo Vanzin <va...@cloudera.com>]

On Tue, Dec 1, 2015 at 12:45 PM, Charles Allen
<ch...@metamarkets.com> wrote:
> Is there a way to pass configuration file resources to be resolvable through
> the classloader?

Not in general. If you're using YARN, you can cheat and use
"spark.yarn.dist.files" which will place those files in the classpath;
the same for "--files" in yarn cluster mode (but *not* client mode!).

-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: ClassLoader resources on executor

[Charles Allen <ch...@metamarkets.com>]

I did some digging and SparkContext.addJar seems to propagate any file into
the classloader on the executors. That seems to have fixed my problem.

It might also be a way to push log4j.properties or log4j2.xml (depending on
what version you are using) down to the executors.

On Wed, Dec 2, 2015 at 12:14 PM Charles Allen <ch...@metamarkets.com>
wrote:

> I still have to propagate the file into the directory somehow, and also
> that's marked as only for legacy jobs (deprecated?), so no, I have not
> experimented with it yet.
>
> On Wed, Dec 2, 2015 at 12:53 AM Rishi Mishra <rm...@snappydata.io>
> wrote:
>
>> Did you try to use *spark.executor.extraClassPath*. The classpath
>> resources will be accessible through the executors class loader which
>> executes your job.
>>
>> On Wed, Dec 2, 2015 at 2:15 AM, Charles Allen <
>> charles.allen@metamarkets.com> wrote:
>>
>>> Is there a way to pass configuration file resources to be resolvable
>>> through the classloader?
>>>
>>> For example, if I'm using a library (non-spark) that can use a
>>> some-lib.properties file in the classpath/classLoader, can I pass that file
>>> so that when it tries to get the resource from the classloader it is able
>>> to find it?
>>>
>>> One potential solution is to take the files and package them as
>>> resources in a jar, and include the jar as part of the spark job, but that
>>> feels like a hack instead of an actual solution.
>>>
>>> Is there any support or planned support for such a thing?
>>>
>>> https://github.com/apache/spark/pull/9118 seems to tackle a similar
>>> problem in a hard-coded kind of way.
>>>
>>> Thank you,
>>> Charles Allen
>>>
>>
>>
>>
>> --
>> Regards,
>> Rishitesh Mishra,
>> SnappyData . (http://www.snappydata.io/)
>>
>> https://in.linkedin.com/in/rishiteshmishra
>>
>

Re: ClassLoader resources on executor

[Charles Allen <ch...@metamarkets.com>]

I still have to propagate the file into the directory somehow, and also
that's marked as only for legacy jobs (deprecated?), so no, I have not
experimented with it yet.

On Wed, Dec 2, 2015 at 12:53 AM Rishi Mishra <rm...@snappydata.io> wrote:

> Did you try to use *spark.executor.extraClassPath*. The classpath
> resources will be accessible through the executors class loader which
> executes your job.
>
> On Wed, Dec 2, 2015 at 2:15 AM, Charles Allen <
> charles.allen@metamarkets.com> wrote:
>
>> Is there a way to pass configuration file resources to be resolvable
>> through the classloader?
>>
>> For example, if I'm using a library (non-spark) that can use a
>> some-lib.properties file in the classpath/classLoader, can I pass that file
>> so that when it tries to get the resource from the classloader it is able
>> to find it?
>>
>> One potential solution is to take the files and package them as resources
>> in a jar, and include the jar as part of the spark job, but that feels like
>> a hack instead of an actual solution.
>>
>> Is there any support or planned support for such a thing?
>>
>> https://github.com/apache/spark/pull/9118 seems to tackle a similar
>> problem in a hard-coded kind of way.
>>
>> Thank you,
>> Charles Allen
>>
>
>
>
> --
> Regards,
> Rishitesh Mishra,
> SnappyData . (http://www.snappydata.io/)
>
> https://in.linkedin.com/in/rishiteshmishra
>

Re: ClassLoader resources on executor

[Rishi Mishra <rm...@snappydata.io>]

Did you try to use *spark.executor.extraClassPath*. The classpath resources
will be accessible through the executors class loader which executes your
job.

On Wed, Dec 2, 2015 at 2:15 AM, Charles Allen <charles.allen@metamarkets.com
> wrote:

> Is there a way to pass configuration file resources to be resolvable
> through the classloader?
>
> For example, if I'm using a library (non-spark) that can use a
> some-lib.properties file in the classpath/classLoader, can I pass that file
> so that when it tries to get the resource from the classloader it is able
> to find it?
>
> One potential solution is to take the files and package them as resources
> in a jar, and include the jar as part of the spark job, but that feels like
> a hack instead of an actual solution.
>
> Is there any support or planned support for such a thing?
>
> https://github.com/apache/spark/pull/9118 seems to tackle a similar
> problem in a hard-coded kind of way.
>
> Thank you,
> Charles Allen
>



-- 
Regards,
Rishitesh Mishra,
SnappyData . (http://www.snappydata.io/)

https://in.linkedin.com/in/rishiteshmishra