You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Arun Lists <li...@gmail.com> on 2015/01/14 03:35:59 UTC

Reading resource files in a Spark application

In some classes, I initialize some values from resource files using the
following snippet:

new File(cls.getResource(fileName).toURI)

This works fine in SBT. When I run it using spark-submit, I get a
bunch of errors because the classes cannot be initialized. What can I
do to make such initialization that is Spark friendly?

Thanks,
arun

Re: Reading resource files in a Spark application

Posted by Arun Lists <li...@gmail.com>.
The problem is that it gives an error message saying something to the
effect that:

URI is not hierarchical

This is consistent with your explanation.

Thanks,
arun


On Wed, Jan 14, 2015 at 1:14 AM, Sean Owen <so...@cloudera.com> wrote:

> My hunch is that it is because the URI of a resource in a JAR file will
> necessarily be specific to where the JAR is on the local filesystem and
> that is not portable or the right way to read a resource. But you didn't
> specify the problem here.
> On Jan 14, 2015 5:15 AM, "Arun Lists" <li...@gmail.com> wrote:
>
>> I experimented with using getResourceAsStream(cls, fileName) instead
>> cls.getResource(fileName).toURI. That works!
>>
>> I have no idea why the latter method does not work in Spark. Any
>> explanations would be welcome.
>>
>> Thanks,
>> arun
>>
>>
>> On Tue, Jan 13, 2015 at 6:35 PM, Arun Lists <li...@gmail.com> wrote:
>>
>>> In some classes, I initialize some values from resource files using the
>>> following snippet:
>>>
>>> new File(cls.getResource(fileName).toURI)
>>>
>>> This works fine in SBT. When I run it using spark-submit, I get a bunch of errors because the classes cannot be initialized. What can I do to make such initialization that is Spark friendly?
>>>
>>> Thanks,
>>> arun
>>>
>>>
>>>
>>

Re: Reading resource files in a Spark application

Posted by Sean Owen <so...@cloudera.com>.
My hunch is that it is because the URI of a resource in a JAR file will
necessarily be specific to where the JAR is on the local filesystem and
that is not portable or the right way to read a resource. But you didn't
specify the problem here.
On Jan 14, 2015 5:15 AM, "Arun Lists" <li...@gmail.com> wrote:

> I experimented with using getResourceAsStream(cls, fileName) instead
> cls.getResource(fileName).toURI. That works!
>
> I have no idea why the latter method does not work in Spark. Any
> explanations would be welcome.
>
> Thanks,
> arun
>
>
> On Tue, Jan 13, 2015 at 6:35 PM, Arun Lists <li...@gmail.com> wrote:
>
>> In some classes, I initialize some values from resource files using the
>> following snippet:
>>
>> new File(cls.getResource(fileName).toURI)
>>
>> This works fine in SBT. When I run it using spark-submit, I get a bunch of errors because the classes cannot be initialized. What can I do to make such initialization that is Spark friendly?
>>
>> Thanks,
>> arun
>>
>>
>>
>

Re: Reading resource files in a Spark application

Posted by Arun Lists <li...@gmail.com>.
I experimented with using getResourceAsStream(cls, fileName) instead
cls.getResource(fileName).toURI. That works!

I have no idea why the latter method does not work in Spark. Any
explanations would be welcome.

Thanks,
arun


On Tue, Jan 13, 2015 at 6:35 PM, Arun Lists <li...@gmail.com> wrote:

> In some classes, I initialize some values from resource files using the
> following snippet:
>
> new File(cls.getResource(fileName).toURI)
>
> This works fine in SBT. When I run it using spark-submit, I get a bunch of errors because the classes cannot be initialized. What can I do to make such initialization that is Spark friendly?
>
> Thanks,
> arun
>
>
>