You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Udit Mehta <um...@groupon.com> on 2015/06/26 00:36:43 UTC
Unable to run spark interpreter on Zeppelin
Hi,
I am facing some issues while running zeppelin spark interpreter in the
yarn client mode. I get the following error in the
zeppelin-interpreter-spark*.log:
ERROR [2015-06-25 22:17:56,374] ({pool-1-thread-4}
ProcessFunction.java[process]:41) - Internal error processing getProgress
org.apache.zeppelin.interpreter.InterpreterException:
java.lang.UnsupportedOperationException: Not implemented by the TFS
FileSystem implementation
at
org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:75)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:109)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:299)
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:938)
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:923)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.UnsupportedOperationException: Not implemented by the
TFS FileSystem implementation
at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at
org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:216)
at
org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:384)
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:102)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:381)
at
org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:276)
at
org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:149)
at
org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:398)
at
org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:73)
... 11 more
Im running the interpreter in yarn-client mode. Here is the command i used
to build the zeppelin jar:
mvn clean install -DskipTests -Pspark-1.3 -Dspark.version=1.3.1
-Phadoop-2.6 -Pyarn
Here is my conf/interpreter.json (spark section):
> "spark.yarn.access.namenodes":
> "hdfs://namenode1.snc1:8032,hdfs://namenode2.snc1:8032",
> "master": "yarn-client",
> "zeppelin.spark.maxResult": "1000",
> "zeppelin.dep.localrepo": "local-repo",
> "spark.app.name": "Zeppelin-umehta",
> "spark.yarn.queue": "public",
> "spark.executor.memory": "512m",
> "zeppelin.spark.useHiveContext": "true",
> "zeppelin.spark.concurrentSQL": "false",
> "spark.home": "/path/to/spark-1.3"
>
And heres my conf/zeppelin-env
> export HADOOP_CONF_DIR=/etc/hadoop/conf
> export ZEPPELIN_PORT=10008
> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.2.0.0-2041"
> export
> ZEPPELIN_INTERPRETER_DIR=/home/umehta/zeppelin-0.6.0-incubating-SNAPSHOT/interpreter
> export
> ZEPPELIN_INTERPRETERS=org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter
>
>
Im using hdp 2.2.0.0-2041 on my cluster.
Does anyone have any insights on this?
Thanks in advance,
Udit
Re: Unable to run spark interpreter on Zeppelin
Posted by Ophir Cohen <op...@gmail.com>.
Did anyone solve that?
I'm still getting it on Spark 1.5
On Sat, Jun 27, 2015 at 5:03 AM, moon soo Lee <mo...@apache.org> wrote:
> That is really strange.
> Could you make sure your core-site.xml does not have anything related
> tachyon?
>
>
> On Fri, Jun 26, 2015 at 4:09 PM Udit Mehta <um...@groupon.com> wrote:
>
>> could this be related: https://issues.apache.org/jira/browse/SPARK-8385 ?
>> Im not sure since im building using spark-1.2 profile.
>>
>> On Thu, Jun 25, 2015 at 11:22 PM, Udit Mehta <um...@groupon.com> wrote:
>>
>>> Hi,
>>>
>>> Thanks for the reply moon. I dont see a reason to add TFS based
>>> libraries when I am not using them. Do you know of any instance when I need
>>> to add them even if I am not using Tachyon?
>>>
>>> Udit
>>>
>>> On Thu, Jun 25, 2015 at 9:13 PM, moon soo Lee <mo...@apache.org> wrote:
>>>
>>>> Hi,
>>>>
>>>> If i guess based on error
>>>>
>>>> Caused by: java.lang.UnsupportedOperationException: Not implemented by
>>>> the TFS FileSystem implementation
>>>>
>>>> It looks like you need add libraries for the TFS FileSystem into the
>>>> classapath of Zeppelin's spark interpreter.
>>>>
>>>> Best,
>>>> moon
>>>>
>>>>
>>>> On Thu, Jun 25, 2015 at 3:36 PM Udit Mehta <um...@groupon.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am facing some issues while running zeppelin spark interpreter in
>>>>> the yarn client mode. I get the following error in the
>>>>> zeppelin-interpreter-spark*.log:
>>>>>
>>>>> ERROR [2015-06-25 22:17:56,374] ({pool-1-thread-4}
>>>>> ProcessFunction.java[process]:41) - Internal error processing getProgress
>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>> java.lang.UnsupportedOperationException: Not implemented by the TFS
>>>>> FileSystem implementation
>>>>> at
>>>>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:75)
>>>>> at
>>>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
>>>>> at
>>>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:109)
>>>>> at
>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:299)
>>>>> at
>>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:938)
>>>>> at
>>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:923)
>>>>> at
>>>>> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>>>>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>>>>> at
>>>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>
>>>>> Caused by: java.lang.UnsupportedOperationException: Not implemented by
>>>>> the TFS FileSystem implementation
>>>>> at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
>>>>> at
>>>>> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
>>>>> at
>>>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
>>>>> at
>>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
>>>>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
>>>>> at
>>>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
>>>>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
>>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
>>>>> at
>>>>> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:216)
>>>>> at
>>>>> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:384)
>>>>> at
>>>>> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:102)
>>>>> at
>>>>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58)
>>>>> at
>>>>> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
>>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:381)
>>>>> at
>>>>> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:276)
>>>>> at
>>>>> org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:149)
>>>>> at
>>>>> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:398)
>>>>> at
>>>>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:73)
>>>>> ... 11 more
>>>>>
>>>>>
>>>>> Im running the interpreter in yarn-client mode. Here is the command i
>>>>> used to build the zeppelin jar:
>>>>>
>>>>> mvn clean install -DskipTests -Pspark-1.3 -Dspark.version=1.3.1
>>>>> -Phadoop-2.6 -Pyarn
>>>>>
>>>>> Here is my conf/interpreter.json (spark section):
>>>>>
>>>>>> "spark.yarn.access.namenodes":
>>>>>> "hdfs://namenode1.snc1:8032,hdfs://namenode2.snc1:8032",
>>>>>> "master": "yarn-client",
>>>>>> "zeppelin.spark.maxResult": "1000",
>>>>>> "zeppelin.dep.localrepo": "local-repo",
>>>>>> "spark.app.name": "Zeppelin-umehta",
>>>>>> "spark.yarn.queue": "public",
>>>>>> "spark.executor.memory": "512m",
>>>>>> "zeppelin.spark.useHiveContext": "true",
>>>>>> "zeppelin.spark.concurrentSQL": "false",
>>>>>> "spark.home": "/path/to/spark-1.3"
>>>>>>
>>>>>
>>>>> And heres my conf/zeppelin-env
>>>>>
>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>> export ZEPPELIN_PORT=10008
>>>>>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.2.0.0-2041"
>>>>>> export
>>>>>> ZEPPELIN_INTERPRETER_DIR=/home/umehta/zeppelin-0.6.0-incubating-SNAPSHOT/interpreter
>>>>>> export
>>>>>> ZEPPELIN_INTERPRETERS=org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter
>>>>>>
>>>>>>
>>>>>
>>>>> Im using hdp 2.2.0.0-2041 on my cluster.
>>>>>
>>>>> Does anyone have any insights on this?
>>>>>
>>>>> Thanks in advance,
>>>>> Udit
>>>>>
>>>>
>>>
>>
Re: Unable to run spark interpreter on Zeppelin
Posted by moon soo Lee <mo...@apache.org>.
That is really strange.
Could you make sure your core-site.xml does not have anything related
tachyon?
On Fri, Jun 26, 2015 at 4:09 PM Udit Mehta <um...@groupon.com> wrote:
> could this be related: https://issues.apache.org/jira/browse/SPARK-8385 ?
> Im not sure since im building using spark-1.2 profile.
>
> On Thu, Jun 25, 2015 at 11:22 PM, Udit Mehta <um...@groupon.com> wrote:
>
>> Hi,
>>
>> Thanks for the reply moon. I dont see a reason to add TFS based libraries
>> when I am not using them. Do you know of any instance when I need to add
>> them even if I am not using Tachyon?
>>
>> Udit
>>
>> On Thu, Jun 25, 2015 at 9:13 PM, moon soo Lee <mo...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> If i guess based on error
>>>
>>> Caused by: java.lang.UnsupportedOperationException: Not implemented by
>>> the TFS FileSystem implementation
>>>
>>> It looks like you need add libraries for the TFS FileSystem into the
>>> classapath of Zeppelin's spark interpreter.
>>>
>>> Best,
>>> moon
>>>
>>>
>>> On Thu, Jun 25, 2015 at 3:36 PM Udit Mehta <um...@groupon.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am facing some issues while running zeppelin spark interpreter in the
>>>> yarn client mode. I get the following error in the
>>>> zeppelin-interpreter-spark*.log:
>>>>
>>>> ERROR [2015-06-25 22:17:56,374] ({pool-1-thread-4}
>>>> ProcessFunction.java[process]:41) - Internal error processing getProgress
>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>> java.lang.UnsupportedOperationException: Not implemented by the TFS
>>>> FileSystem implementation
>>>> at
>>>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:75)
>>>> at
>>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
>>>> at
>>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:109)
>>>> at
>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:299)
>>>> at
>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:938)
>>>> at
>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:923)
>>>> at
>>>> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>>>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>>>> at
>>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>>
>>>> Caused by: java.lang.UnsupportedOperationException: Not implemented by
>>>> the TFS FileSystem implementation
>>>> at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
>>>> at
>>>> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
>>>> at
>>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
>>>> at
>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
>>>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
>>>> at
>>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
>>>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
>>>> at
>>>> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:216)
>>>> at
>>>> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:384)
>>>> at
>>>> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:102)
>>>> at
>>>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58)
>>>> at
>>>> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:381)
>>>> at
>>>> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:276)
>>>> at
>>>> org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:149)
>>>> at
>>>> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:398)
>>>> at
>>>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:73)
>>>> ... 11 more
>>>>
>>>>
>>>> Im running the interpreter in yarn-client mode. Here is the command i
>>>> used to build the zeppelin jar:
>>>>
>>>> mvn clean install -DskipTests -Pspark-1.3 -Dspark.version=1.3.1
>>>> -Phadoop-2.6 -Pyarn
>>>>
>>>> Here is my conf/interpreter.json (spark section):
>>>>
>>>>> "spark.yarn.access.namenodes":
>>>>> "hdfs://namenode1.snc1:8032,hdfs://namenode2.snc1:8032",
>>>>> "master": "yarn-client",
>>>>> "zeppelin.spark.maxResult": "1000",
>>>>> "zeppelin.dep.localrepo": "local-repo",
>>>>> "spark.app.name": "Zeppelin-umehta",
>>>>> "spark.yarn.queue": "public",
>>>>> "spark.executor.memory": "512m",
>>>>> "zeppelin.spark.useHiveContext": "true",
>>>>> "zeppelin.spark.concurrentSQL": "false",
>>>>> "spark.home": "/path/to/spark-1.3"
>>>>>
>>>>
>>>> And heres my conf/zeppelin-env
>>>>
>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>> export ZEPPELIN_PORT=10008
>>>>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.2.0.0-2041"
>>>>> export
>>>>> ZEPPELIN_INTERPRETER_DIR=/home/umehta/zeppelin-0.6.0-incubating-SNAPSHOT/interpreter
>>>>> export
>>>>> ZEPPELIN_INTERPRETERS=org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter
>>>>>
>>>>>
>>>>
>>>> Im using hdp 2.2.0.0-2041 on my cluster.
>>>>
>>>> Does anyone have any insights on this?
>>>>
>>>> Thanks in advance,
>>>> Udit
>>>>
>>>
>>
>
Re: Unable to run spark interpreter on Zeppelin
Posted by Udit Mehta <um...@groupon.com>.
could this be related: https://issues.apache.org/jira/browse/SPARK-8385 ?
Im not sure since im building using spark-1.2 profile.
On Thu, Jun 25, 2015 at 11:22 PM, Udit Mehta <um...@groupon.com> wrote:
> Hi,
>
> Thanks for the reply moon. I dont see a reason to add TFS based libraries
> when I am not using them. Do you know of any instance when I need to add
> them even if I am not using Tachyon?
>
> Udit
>
> On Thu, Jun 25, 2015 at 9:13 PM, moon soo Lee <mo...@apache.org> wrote:
>
>> Hi,
>>
>> If i guess based on error
>>
>> Caused by: java.lang.UnsupportedOperationException: Not implemented by
>> the TFS FileSystem implementation
>>
>> It looks like you need add libraries for the TFS FileSystem into the
>> classapath of Zeppelin's spark interpreter.
>>
>> Best,
>> moon
>>
>>
>> On Thu, Jun 25, 2015 at 3:36 PM Udit Mehta <um...@groupon.com> wrote:
>>
>>> Hi,
>>>
>>> I am facing some issues while running zeppelin spark interpreter in the
>>> yarn client mode. I get the following error in the
>>> zeppelin-interpreter-spark*.log:
>>>
>>> ERROR [2015-06-25 22:17:56,374] ({pool-1-thread-4}
>>> ProcessFunction.java[process]:41) - Internal error processing getProgress
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> java.lang.UnsupportedOperationException: Not implemented by the TFS
>>> FileSystem implementation
>>> at
>>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:75)
>>> at
>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
>>> at
>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:109)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:299)
>>> at
>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:938)
>>> at
>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:923)
>>> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>>> at
>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>>
>>> Caused by: java.lang.UnsupportedOperationException: Not implemented by
>>> the TFS FileSystem implementation
>>> at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
>>> at
>>> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
>>> at
>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
>>> at
>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
>>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
>>> at
>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
>>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
>>> at
>>> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:216)
>>> at
>>> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:384)
>>> at
>>> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:102)
>>> at
>>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58)
>>> at
>>> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:381)
>>> at
>>> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:276)
>>> at
>>> org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:149)
>>> at
>>> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:398)
>>> at
>>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:73)
>>> ... 11 more
>>>
>>>
>>> Im running the interpreter in yarn-client mode. Here is the command i
>>> used to build the zeppelin jar:
>>>
>>> mvn clean install -DskipTests -Pspark-1.3 -Dspark.version=1.3.1
>>> -Phadoop-2.6 -Pyarn
>>>
>>> Here is my conf/interpreter.json (spark section):
>>>
>>>> "spark.yarn.access.namenodes":
>>>> "hdfs://namenode1.snc1:8032,hdfs://namenode2.snc1:8032",
>>>> "master": "yarn-client",
>>>> "zeppelin.spark.maxResult": "1000",
>>>> "zeppelin.dep.localrepo": "local-repo",
>>>> "spark.app.name": "Zeppelin-umehta",
>>>> "spark.yarn.queue": "public",
>>>> "spark.executor.memory": "512m",
>>>> "zeppelin.spark.useHiveContext": "true",
>>>> "zeppelin.spark.concurrentSQL": "false",
>>>> "spark.home": "/path/to/spark-1.3"
>>>>
>>>
>>> And heres my conf/zeppelin-env
>>>
>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>> export ZEPPELIN_PORT=10008
>>>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.2.0.0-2041"
>>>> export
>>>> ZEPPELIN_INTERPRETER_DIR=/home/umehta/zeppelin-0.6.0-incubating-SNAPSHOT/interpreter
>>>> export
>>>> ZEPPELIN_INTERPRETERS=org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter
>>>>
>>>>
>>>
>>> Im using hdp 2.2.0.0-2041 on my cluster.
>>>
>>> Does anyone have any insights on this?
>>>
>>> Thanks in advance,
>>> Udit
>>>
>>
>
Re: Unable to run spark interpreter on Zeppelin
Posted by Udit Mehta <um...@groupon.com>.
Hi,
Thanks for the reply moon. I dont see a reason to add TFS based libraries
when I am not using them. Do you know of any instance when I need to add
them even if I am not using Tachyon?
Udit
On Thu, Jun 25, 2015 at 9:13 PM, moon soo Lee <mo...@apache.org> wrote:
> Hi,
>
> If i guess based on error
>
> Caused by: java.lang.UnsupportedOperationException: Not implemented by the
> TFS FileSystem implementation
>
> It looks like you need add libraries for the TFS FileSystem into the
> classapath of Zeppelin's spark interpreter.
>
> Best,
> moon
>
>
> On Thu, Jun 25, 2015 at 3:36 PM Udit Mehta <um...@groupon.com> wrote:
>
>> Hi,
>>
>> I am facing some issues while running zeppelin spark interpreter in the
>> yarn client mode. I get the following error in the
>> zeppelin-interpreter-spark*.log:
>>
>> ERROR [2015-06-25 22:17:56,374] ({pool-1-thread-4}
>> ProcessFunction.java[process]:41) - Internal error processing getProgress
>> org.apache.zeppelin.interpreter.InterpreterException:
>> java.lang.UnsupportedOperationException: Not implemented by the TFS
>> FileSystem implementation
>> at
>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:75)
>> at
>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
>> at
>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:109)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:299)
>> at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:938)
>> at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:923)
>> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>> at
>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>>
>> Caused by: java.lang.UnsupportedOperationException: Not implemented by
>> the TFS FileSystem implementation
>> at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
>> at
>> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
>> at
>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
>> at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
>> at
>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
>> at
>> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:216)
>> at
>> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:384)
>> at
>> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:102)
>> at
>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58)
>> at
>> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:381)
>> at
>> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:276)
>> at
>> org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:149)
>> at
>> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:398)
>> at
>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:73)
>> ... 11 more
>>
>>
>> Im running the interpreter in yarn-client mode. Here is the command i
>> used to build the zeppelin jar:
>>
>> mvn clean install -DskipTests -Pspark-1.3 -Dspark.version=1.3.1
>> -Phadoop-2.6 -Pyarn
>>
>> Here is my conf/interpreter.json (spark section):
>>
>>> "spark.yarn.access.namenodes":
>>> "hdfs://namenode1.snc1:8032,hdfs://namenode2.snc1:8032",
>>> "master": "yarn-client",
>>> "zeppelin.spark.maxResult": "1000",
>>> "zeppelin.dep.localrepo": "local-repo",
>>> "spark.app.name": "Zeppelin-umehta",
>>> "spark.yarn.queue": "public",
>>> "spark.executor.memory": "512m",
>>> "zeppelin.spark.useHiveContext": "true",
>>> "zeppelin.spark.concurrentSQL": "false",
>>> "spark.home": "/path/to/spark-1.3"
>>>
>>
>> And heres my conf/zeppelin-env
>>
>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>> export ZEPPELIN_PORT=10008
>>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.2.0.0-2041"
>>> export
>>> ZEPPELIN_INTERPRETER_DIR=/home/umehta/zeppelin-0.6.0-incubating-SNAPSHOT/interpreter
>>> export
>>> ZEPPELIN_INTERPRETERS=org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter
>>>
>>>
>>
>> Im using hdp 2.2.0.0-2041 on my cluster.
>>
>> Does anyone have any insights on this?
>>
>> Thanks in advance,
>> Udit
>>
>
Re: Unable to run spark interpreter on Zeppelin
Posted by moon soo Lee <mo...@apache.org>.
Hi,
If i guess based on error
Caused by: java.lang.UnsupportedOperationException: Not implemented by the
TFS FileSystem implementation
It looks like you need add libraries for the TFS FileSystem into the
classapath of Zeppelin's spark interpreter.
Best,
moon
On Thu, Jun 25, 2015 at 3:36 PM Udit Mehta <um...@groupon.com> wrote:
> Hi,
>
> I am facing some issues while running zeppelin spark interpreter in the
> yarn client mode. I get the following error in the
> zeppelin-interpreter-spark*.log:
>
> ERROR [2015-06-25 22:17:56,374] ({pool-1-thread-4}
> ProcessFunction.java[process]:41) - Internal error processing getProgress
> org.apache.zeppelin.interpreter.InterpreterException:
> java.lang.UnsupportedOperationException: Not implemented by the TFS
> FileSystem implementation
> at
> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:75)
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:109)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:299)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:938)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:923)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
> Caused by: java.lang.UnsupportedOperationException: Not implemented by the
> TFS FileSystem implementation
> at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
> at
> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
> at
> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:216)
> at
> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:384)
> at
> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:102)
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:381)
> at
> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:276)
> at
> org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:149)
> at
> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:398)
> at
> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:73)
> ... 11 more
>
>
> Im running the interpreter in yarn-client mode. Here is the command i used
> to build the zeppelin jar:
>
> mvn clean install -DskipTests -Pspark-1.3 -Dspark.version=1.3.1
> -Phadoop-2.6 -Pyarn
>
> Here is my conf/interpreter.json (spark section):
>
>> "spark.yarn.access.namenodes":
>> "hdfs://namenode1.snc1:8032,hdfs://namenode2.snc1:8032",
>> "master": "yarn-client",
>> "zeppelin.spark.maxResult": "1000",
>> "zeppelin.dep.localrepo": "local-repo",
>> "spark.app.name": "Zeppelin-umehta",
>> "spark.yarn.queue": "public",
>> "spark.executor.memory": "512m",
>> "zeppelin.spark.useHiveContext": "true",
>> "zeppelin.spark.concurrentSQL": "false",
>> "spark.home": "/path/to/spark-1.3"
>>
>
> And heres my conf/zeppelin-env
>
>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>> export ZEPPELIN_PORT=10008
>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.2.0.0-2041"
>> export
>> ZEPPELIN_INTERPRETER_DIR=/home/umehta/zeppelin-0.6.0-incubating-SNAPSHOT/interpreter
>> export
>> ZEPPELIN_INTERPRETERS=org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter
>>
>>
>
> Im using hdp 2.2.0.0-2041 on my cluster.
>
> Does anyone have any insights on this?
>
> Thanks in advance,
> Udit
>