Re: Unable to run spark interpreter on Zeppelin

[Ophir Cohen <op...@gmail.com>]

Did anyone solve that?
I'm still getting it on Spark 1.5

On Sat, Jun 27, 2015 at 5:03 AM, moon soo Lee <mo...@apache.org> wrote:

> That is really strange.
> Could you make sure your core-site.xml does not have anything related
> tachyon?
>
>
> On Fri, Jun 26, 2015 at 4:09 PM Udit Mehta <um...@groupon.com> wrote:
>
>> could this be related: https://issues.apache.org/jira/browse/SPARK-8385 ?
>> Im not sure since im building using spark-1.2 profile.
>>
>> On Thu, Jun 25, 2015 at 11:22 PM, Udit Mehta <um...@groupon.com> wrote:
>>
>>> Hi,
>>>
>>> Thanks for the reply moon. I dont see a reason to add TFS based
>>> libraries when I am not using them. Do you know of any instance when I need
>>> to add them even if I am not using Tachyon?
>>>
>>> Udit
>>>
>>> On Thu, Jun 25, 2015 at 9:13 PM, moon soo Lee <mo...@apache.org> wrote:
>>>
>>>> Hi,
>>>>
>>>> If i guess based on error
>>>>
>>>> Caused by: java.lang.UnsupportedOperationException: Not implemented by
>>>> the TFS FileSystem implementation
>>>>
>>>> It looks like you need add libraries for the TFS FileSystem into the
>>>> classapath of Zeppelin's spark interpreter.
>>>>
>>>> Best,
>>>> moon
>>>>
>>>>
>>>> On Thu, Jun 25, 2015 at 3:36 PM Udit Mehta <um...@groupon.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am facing some issues while running zeppelin spark interpreter in
>>>>> the yarn client mode. I get the following error in the
>>>>> zeppelin-interpreter-spark*.log:
>>>>>
>>>>> ERROR [2015-06-25 22:17:56,374] ({pool-1-thread-4}
>>>>> ProcessFunction.java[process]:41) - Internal error processing getProgress
>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>> java.lang.UnsupportedOperationException: Not implemented by the TFS
>>>>> FileSystem implementation
>>>>>     at
>>>>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:75)
>>>>>     at
>>>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
>>>>>     at
>>>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:109)
>>>>>     at
>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:299)
>>>>>     at
>>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:938)
>>>>>     at
>>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:923)
>>>>>     at
>>>>> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>>>>>     at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>>>>>     at
>>>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
>>>>>     at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>     at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>     at java.lang.Thread.run(Thread.java:744)
>>>>>
>>>>> Caused by: java.lang.UnsupportedOperationException: Not implemented by
>>>>> the TFS FileSystem implementation
>>>>>     at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
>>>>>     at
>>>>> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
>>>>>     at
>>>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
>>>>>     at
>>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
>>>>>     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
>>>>>     at
>>>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
>>>>>     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
>>>>>     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>>>>>     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
>>>>>     at
>>>>> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:216)
>>>>>     at
>>>>> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:384)
>>>>>     at
>>>>> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:102)
>>>>>     at
>>>>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58)
>>>>>     at
>>>>> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
>>>>>     at org.apache.spark.SparkContext.<init>(SparkContext.scala:381)
>>>>>     at
>>>>> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:276)
>>>>>     at
>>>>> org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:149)
>>>>>     at
>>>>> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:398)
>>>>>     at
>>>>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:73)
>>>>>     ... 11 more
>>>>>
>>>>>
>>>>> Im running the interpreter in yarn-client mode. Here is the command i
>>>>> used to build the zeppelin jar:
>>>>>
>>>>> mvn clean install -DskipTests -Pspark-1.3 -Dspark.version=1.3.1
>>>>> -Phadoop-2.6 -Pyarn
>>>>>
>>>>> Here is my conf/interpreter.json (spark section):
>>>>>
>>>>>>         "spark.yarn.access.namenodes":
>>>>>> "hdfs://namenode1.snc1:8032,hdfs://namenode2.snc1:8032",
>>>>>>         "master": "yarn-client",
>>>>>>         "zeppelin.spark.maxResult": "1000",
>>>>>>         "zeppelin.dep.localrepo": "local-repo",
>>>>>>         "spark.app.name": "Zeppelin-umehta",
>>>>>>         "spark.yarn.queue": "public",
>>>>>>         "spark.executor.memory": "512m",
>>>>>>         "zeppelin.spark.useHiveContext": "true",
>>>>>>         "zeppelin.spark.concurrentSQL": "false",
>>>>>>         "spark.home": "/path/to/spark-1.3"
>>>>>>
>>>>>
>>>>> And heres my conf/zeppelin-env
>>>>>
>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>> export ZEPPELIN_PORT=10008
>>>>>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.2.0.0-2041"
>>>>>> export
>>>>>> ZEPPELIN_INTERPRETER_DIR=/home/umehta/zeppelin-0.6.0-incubating-SNAPSHOT/interpreter
>>>>>> export
>>>>>> ZEPPELIN_INTERPRETERS=org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter
>>>>>>
>>>>>>
>>>>>
>>>>> Im using hdp 2.2.0.0-2041 on my cluster.
>>>>>
>>>>> Does anyone have any insights on this?
>>>>>
>>>>> Thanks in advance,
>>>>> Udit
>>>>>
>>>>
>>>
>>