You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Daniel Haviv <da...@veracity-group.com> on 2015/07/02 16:38:18 UTC
thrift-server does not load jars files (Azure HDInsight)
Hi,
I'm trying to start the thrift-server and passing it azure's blob storage
jars but I'm failing on :
Caused by: java.io.IOException: No FileSystem for scheme: wasb
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:342)
... 16 more
If I start the spark-shell the same way, everything works fine.
spark-shell command:
./bin/spark-shell --master yarn --jars
/home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
--num-executors 4
thrift-server command:
./sbin/start-thriftserver.sh --master yarn--jars
/home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
--num-executors 4
How can I pass dependency jars to the thrift server?
Thanks,
Daniel
Re: thrift-server does not load jars files (Azure HDInsight)
Posted by Daniel Haviv <da...@veracity-group.com>.
Hi,
Just updating back that setting spark.driver.extraClassPath worked.
Thanks,
Daniel
On Fri, Jul 3, 2015 at 5:35 PM, Ted Yu <yu...@gmail.com> wrote:
> Alternatively, setting spark.driver.extraClassPath should work.
>
> Cheers
>
> On Fri, Jul 3, 2015 at 2:59 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>>
>>> On Thu, Jul 2, 2015 at 7:38 AM, Daniel Haviv <
>>> daniel.haviv@veracity-group.com> wrote:
>>>
>>>> Hi,
>>>> I'm trying to start the thrift-server and passing it azure's blob
>>>> storage jars but I'm failing on :
>>>> Caused by: java.io.IOException: No FileSystem for scheme: wasb
>>>> at
>>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
>>>> at
>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
>>>> at
>>>> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
>>>> at
>>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
>>>> at
>>>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
>>>> at
>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:342)
>>>> ... 16 more
>>>>
>>>> If I start the spark-shell the same way, everything works fine.
>>>>
>>>> spark-shell command:
>>>> ./bin/spark-shell --master yarn --jars
>>>> /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
>>>> --num-executors 4
>>>>
>>>> thrift-server command:
>>>> ./sbin/start-thriftserver.sh --master yarn--jars
>>>> /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
>>>> --num-executors 4
>>>>
>>>> How can I pass dependency jars to the thrift server?
>>>>
>>>> Thanks,
>>>> Daniel
>>>>
>>>
>>>
>>
>> you should be able to add the JARs to the environment variable
>> SPARK_SUBMIT_CLASSPATH or SPARK_CLASSPATH and have them picked up when
>> bin/compute-classpath.{cmd.sh} builds up the classpath
>>
>>
>>
>
Re: thrift-server does not load jars files (Azure HDInsight)
Posted by Ted Yu <yu...@gmail.com>.
Alternatively, setting spark.driver.extraClassPath should work.
Cheers
On Fri, Jul 3, 2015 at 2:59 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
>> On Thu, Jul 2, 2015 at 7:38 AM, Daniel Haviv <
>> daniel.haviv@veracity-group.com> wrote:
>>
>>> Hi,
>>> I'm trying to start the thrift-server and passing it azure's blob
>>> storage jars but I'm failing on :
>>> Caused by: java.io.IOException: No FileSystem for scheme: wasb
>>> at
>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
>>> at
>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
>>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
>>> at
>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
>>> at
>>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
>>> at
>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:342)
>>> ... 16 more
>>>
>>> If I start the spark-shell the same way, everything works fine.
>>>
>>> spark-shell command:
>>> ./bin/spark-shell --master yarn --jars
>>> /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
>>> --num-executors 4
>>>
>>> thrift-server command:
>>> ./sbin/start-thriftserver.sh --master yarn--jars
>>> /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
>>> --num-executors 4
>>>
>>> How can I pass dependency jars to the thrift server?
>>>
>>> Thanks,
>>> Daniel
>>>
>>
>>
>
> you should be able to add the JARs to the environment variable
> SPARK_SUBMIT_CLASSPATH or SPARK_CLASSPATH and have them picked up when
> bin/compute-classpath.{cmd.sh} builds up the classpath
>
>
>
Re: thrift-server does not load jars files (Azure HDInsight)
Posted by Steve Loughran <st...@hortonworks.com>.
On Thu, Jul 2, 2015 at 7:38 AM, Daniel Haviv <da...@veracity-group.com>> wrote:
Hi,
I'm trying to start the thrift-server and passing it azure's blob storage jars but I'm failing on :
Caused by: java.io.IOException: No FileSystem for scheme: wasb
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:342)
... 16 more
If I start the spark-shell the same way, everything works fine.
spark-shell command:
./bin/spark-shell --master yarn --jars /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar --num-executors 4
thrift-server command:
./sbin/start-thriftserver.sh --master yarn--jars /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar --num-executors 4
How can I pass dependency jars to the thrift server?
Thanks,
Daniel
you should be able to add the JARs to the environment variable SPARK_SUBMIT_CLASSPATH or SPARK_CLASSPATH and have them picked up when bin/compute-classpath.{cmd.sh} builds up the classpath
Re: thrift-server does not load jars files (Azure HDInsight)
Posted by Daniel Haviv <da...@veracity-group.com>.
Hi,
I'm using 1.4.
It's indeed a typo in the email itself.
Thanks,
Daniel
On Thu, Jul 2, 2015 at 6:06 PM, Ted Yu <yu...@gmail.com> wrote:
> Which Spark release are you using ?
>
> bq. yarn--jars
>
> I guess the above was just a typo in your email (missing space).
>
> Cheers
>
> On Thu, Jul 2, 2015 at 7:38 AM, Daniel Haviv <
> daniel.haviv@veracity-group.com> wrote:
>
>> Hi,
>> I'm trying to start the thrift-server and passing it azure's blob storage
>> jars but I'm failing on :
>> Caused by: java.io.IOException: No FileSystem for scheme: wasb
>> at
>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
>> at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
>> at
>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
>> at
>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:342)
>> ... 16 more
>>
>> If I start the spark-shell the same way, everything works fine.
>>
>> spark-shell command:
>> ./bin/spark-shell --master yarn --jars
>> /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
>> --num-executors 4
>>
>> thrift-server command:
>> ./sbin/start-thriftserver.sh --master yarn--jars
>> /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
>> --num-executors 4
>>
>> How can I pass dependency jars to the thrift server?
>>
>> Thanks,
>> Daniel
>>
>
>
Re: thrift-server does not load jars files (Azure HDInsight)
Posted by Ted Yu <yu...@gmail.com>.
Which Spark release are you using ?
bq. yarn--jars
I guess the above was just a typo in your email (missing space).
Cheers
On Thu, Jul 2, 2015 at 7:38 AM, Daniel Haviv <
daniel.haviv@veracity-group.com> wrote:
> Hi,
> I'm trying to start the thrift-server and passing it azure's blob storage
> jars but I'm failing on :
> Caused by: java.io.IOException: No FileSystem for scheme: wasb
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:342)
> ... 16 more
>
> If I start the spark-shell the same way, everything works fine.
>
> spark-shell command:
> ./bin/spark-shell --master yarn --jars
> /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
> --num-executors 4
>
> thrift-server command:
> ./sbin/start-thriftserver.sh --master yarn--jars
> /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
> --num-executors 4
>
> How can I pass dependency jars to the thrift server?
>
> Thanks,
> Daniel
>