You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by 江之源 <ji...@liulishuo.com> on 2015/07/21 04:35:42 UTC
a small question about zeppelin-manager
hi
i installed zeppelin some time before, but it always failed in my server
cluster. i found the z-management Occasionally. I installed and success in
my server. But when i wanna to read in my HDFS file like:
sc.textFile("hdfs://llscluster/tmp/jzyresult/part-04093").count()
it throw the errors in my cluster:Job aborted due to stage failure: Task 15
in stage 6.0 failed 4 times, most recent failure: Lost task 15.3 in stage
6.0 (TID 386, lls7): java.io.EOFException
when i modify it to the local model, it could read HDFS file successfully.
My cluster is Spark1.3.0 Hadoop2.0.0-CDH4.5.0. but the install options just
have Spark1.3.0 and Hadoop2.0.0-CDH-4.7.0. Is this the cause to read HDFS
file failed?
Look forward to your reply!
THANK YOU!
JZY
Re: a small question about zeppelin-manager
Posted by 江之源 <ji...@liulishuo.com>.
i know i just wanna to test the shell interpreter's function. and i am
confused....
On Wed, Jul 22, 2015 at 9:02 PM, IT CTO <go...@gmail.com> wrote:
> Why would you want to run spark-shell from zeppelin? as I see it
> spark-shell in a REPL tool for spark and zeppelin is a REPL tool for spark
> (and others...)
> Eran
>
> On Wed, Jul 22, 2015 at 3:47 PM 江之源 <ji...@liulishuo.com> wrote:
>
>> ok it works! 3q
>> And i have an another question about shell
>> The shell could work when type some simple commands.but it couldn’t work
>> when i wanna to start a spark shell or spark submit even the sudo command.
>> I wanna know the shell interpreter’s limit. why it always call
>> Process exited with an error: 1 (Exit value: 1) when i type sudo or
>> spark-submit
>>
>> and when i type spark-shell it feedback:
>>
>>
>> setting ulimit -m to 57671680 Spark assembly has been built with Hive,
>> including Datanucleus jars on classpath SLF4J: Class path contains multiple
>> SLF4J bindings. SLF4J: Found binding in
>> [jar:file:/home/pipeline/zeppelin-manager/zeppelin-0.5.0-incubating-SNAPSHOT/interpreter/sh/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in
>> [jar:file:/home/pipeline/zeppelin-manager/zeppelin-0.5.0-incubating-SNAPSHOT/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in
>> [jar:file:/opt/simple-spark.1.3.0/lib/spark-assembly-1.3.0-hadoop2.0.0-mr1-cdh4.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>> explanation. SLF4J: Actual binding is of type
>> [org.slf4j.impl.Log4jLoggerFactory] log4j:ERROR setFile(null,true) call
>> failed. java.io.FileNotFoundException: (没有那个文件或目录) at
>> java.io.FileOutputStream.open(Native Method) at
>> java.io.FileOutputStream.<init>(FileOutputStream.java:221) at
>> java.io.FileOutputStream.<init>(FileOutputStream.java:142) at
>> org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at
>> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165) at
>> org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
>> at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
>> at
>> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
>> at
>> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
>> at
>> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
>> at
>> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
>> at
>> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
>> at
>> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
>> at
>> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
>> at
>> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
>> at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at
>> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64) at
>> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285) at
>> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
>> at
>> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
>> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685) at
>> org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:78)
>> at
>> org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:1996)
>> at
>> org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:1996)
>> at scala.Option.getOrElse(Option.scala:120) at
>> org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:1996) at
>> org.apache.spark.SecurityManager.<init>(SecurityManager.scala:207) at
>> org.apache.spark.repl.SparkIMain.<init>(SparkIMain.scala:118) at
>> org.apache.spark.repl.SparkILoop$SparkILoopInterpreter.<init>(SparkILoop.scala:187)
>> at org.apache.spark.repl.SparkILoop.createInterpreter(SparkILoop.scala:216)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:948)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>> at
>> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>> at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058) at
>> org.apache.spark.repl.Main$.main(Main.scala:31) at
>> org.apache.spark.repl.Main.main(Main.scala) at
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606) at
>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) at
>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) at
>> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) log4j:ERROR
>> Either File or DatePattern options are not set for appender [dailyfile].
>> Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/
>> .__/\_,_/_/ /_/\_\ version 1.3.0 /_/ Using Scala version 2.10.4 (OpenJDK
>> 64-Bit Server VM, Java 1.7.0_71) Type in expressions to have them
>> evaluated. Type :help for more information.
>> org.apache.spark.SparkException: Found both spark.executor.extraClassPath
>> and SPARK_CLASSPATH. Use only the former. at
>> org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$7.apply(SparkConf.scala:339)
>> at
>> org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$7.apply(SparkConf.scala:337)
>> at scala.collection.immutable.List.foreach(List.scala:318) at
>> org.apache.spark.SparkConf$$anonfun$validateSettings$6.apply(SparkConf.scala:337)
>> at
>> org.apache.spark.SparkConf$$anonfun$validateSettings$6.apply(SparkConf.scala:325)
>> at scala.Option.foreach(Option.scala:236) at
>> org.apache.spark.SparkConf.validateSettings(SparkConf.scala:325) at
>> org.apache.spark.SparkContext.<init>(SparkContext.scala:197) at
>> org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1016)
>> at $iwC$$iwC.<init>(<console>:9) at $iwC.<init>(<console>:18) at
>> <init>(<console>:20) at .<init>(<console>:24) at .<clinit>(<console>) at
>> .<init>(<console>:7) at .<clinit>(<console>) at $print(<console>) at
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606) at
>> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>> at
>> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
>> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) at
>> org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) at
>> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856) at
>> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
>> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813) at
>> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:123)
>> at
>> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:122)
>> at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
>> at
>> org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:122)
>> at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64) at
>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:973)
>> at
>> org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:157)
>> at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64) at
>> org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:106)
>> at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:990)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>> at
>> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>> at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058) at
>> org.apache.spark.repl.Main$.main(Main.scala:31) at
>> org.apache.spark.repl.Main.main(Main.scala) at
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606) at
>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) at
>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) at
>> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>> java.lang.NullPointerException at
>> org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:141) at
>> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:49) at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at
>> org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1027) at
>> $iwC$$iwC.<init>(<console>:9)
>> <console>:10: error: not found: value sqlContext import
>> sqlContext.implicits._ ^ <console>:10: error: not found: value sqlContext
>> import sqlContext.sql ^ scala> Stopping spark context. <console>:14: error:
>> not found: value sc sc.stop() ^
>>
>> On Jul 22, 2015, at 8:36 PM, Alexander Bezzubov <ab...@nflabs.com>
>> wrote:
>>
>> Hi,
>>
>> those are very valid steps indeed and you do not need to build custom
>> spark version.
>>
>> Hope this helps!
>>
>> On Tue, Jul 21, 2015 at 7:46 PM, 江之源 <ji...@liulishuo.com> wrote:
>>
>> hi
>> To install the zeppelin with z-manager is a helpless method, because i
>> install the zeppelin in manually way and failed.I have tried many many
>> times.
>> My cluster is spark1.3.0 hadoop 2.0.0-cdh4.5.0,and the model is
>> standalone.
>> I will install the zeppelin manually right now so i wanna you check my
>> steps:
>>
>> 1. git clone the repository from the github.
>> 2. mvn clean package
>> 3. mvn install -DskipTests -Dspark.version=1.3.0
>> -Dhadoop.version=2.0.0-cdh4.5.0
>> (did zeppelin support cdh4.5.0)
>> Should i have to do the custom built spark
>> like(-Dspark.version=1.1.0-Custom)
>> 4.modify my master spark://...:7077
>> Is it completed? or i lost something please tell me.
>> thanks
>> jzy
>>
>> 在 2015年7月21日，下午5:48，Alexander Bezzubov <ab...@nflabs.com> 写道：
>>
>> Hi,
>>
>> thank you for your interest in the project!
>>
>> It seems like the best way to get Zeppelin up and running in your case
>> would be to build it manually with relevant Spark\Hadoop options as
>> described here
>> http://zeppelin.incubator.apache.org/docs/install/install.html
>>
>> Please, let me know if that helps.
>>
>> --
>> BR,
>> Alex
>>
>> On Tue, Jul 21, 2015 at 11:35 AM, 江之源 <ji...@liulishuo.com> wrote:
>>
>> hi
>> i installed zeppelin some time before, but it always failed in my server
>> cluster. i found the z-management Occasionally. I installed and success in
>> my server. But when i wanna to read in my HDFS file like:
>>
>> sc.textFile("hdfs://llscluster/tmp/jzyresult/part-04093").count()
>>
>>
>> it throw the errors in my cluster:Job aborted due to stage failure: Task
>> 15
>> in stage 6.0 failed 4 times, most recent failure: Lost task 15.3 in stage
>> 6.0 (TID 386, lls7): java.io.EOFException
>>
>> when i modify it to the local model, it could read HDFS file successfully.
>> My cluster is Spark1.3.0 Hadoop2.0.0-CDH4.5.0. but the install options
>> just
>> have Spark1.3.0 and Hadoop2.0.0-CDH-4.7.0. Is this the cause to read HDFS
>> file failed?
>> Look forward to your reply!
>> THANK YOU!
>> JZY
>>
>>
>>
>>
>> --
>> --
>> Kind regards,
>> Alexander.
>>
>>
>>
>>
>>
>> --
>> --
>> Kind regards,
>> Alexander.
>>
>>
>>
Re: a small question about zeppelin-manager
Posted by IT CTO <go...@gmail.com>.
Why would you want to run spark-shell from zeppelin? as I see it
spark-shell in a REPL tool for spark and zeppelin is a REPL tool for spark
(and others...)
Eran
On Wed, Jul 22, 2015 at 3:47 PM 江之源 <ji...@liulishuo.com> wrote:
> ok it works! 3q
> And i have an another question about shell
> The shell could work when type some simple commands.but it couldn’t work
> when i wanna to start a spark shell or spark submit even the sudo command.
> I wanna know the shell interpreter’s limit. why it always call
> Process exited with an error: 1 (Exit value: 1) when i type sudo or
> spark-submit
>
> and when i type spark-shell it feedback:
>
>
> setting ulimit -m to 57671680 Spark assembly has been built with Hive,
> including Datanucleus jars on classpath SLF4J: Class path contains multiple
> SLF4J bindings. SLF4J: Found binding in
> [jar:file:/home/pipeline/zeppelin-manager/zeppelin-0.5.0-incubating-SNAPSHOT/interpreter/sh/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/pipeline/zeppelin-manager/zeppelin-0.5.0-incubating-SNAPSHOT/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/simple-spark.1.3.0/lib/spark-assembly-1.3.0-hadoop2.0.0-mr1-cdh4.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation. SLF4J: Actual binding is of type
> [org.slf4j.impl.Log4jLoggerFactory] log4j:ERROR setFile(null,true) call
> failed. java.io.FileNotFoundException: (没有那个文件或目录) at
> java.io.FileOutputStream.open(Native Method) at
> java.io.FileOutputStream.<init>(FileOutputStream.java:221) at
> java.io.FileOutputStream.<init>(FileOutputStream.java:142) at
> org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165) at
> org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
> at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
> at
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64) at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285) at
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
> at
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685) at
> org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:78)
> at
> org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:1996)
> at
> org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:1996)
> at scala.Option.getOrElse(Option.scala:120) at
> org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:1996) at
> org.apache.spark.SecurityManager.<init>(SecurityManager.scala:207) at
> org.apache.spark.repl.SparkIMain.<init>(SparkIMain.scala:118) at
> org.apache.spark.repl.SparkILoop$SparkILoopInterpreter.<init>(SparkILoop.scala:187)
> at org.apache.spark.repl.SparkILoop.createInterpreter(SparkILoop.scala:216)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:948)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> at
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058) at
> org.apache.spark.repl.Main$.main(Main.scala:31) at
> org.apache.spark.repl.Main.main(Main.scala) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) at
> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) log4j:ERROR
> Either File or DatePattern options are not set for appender [dailyfile].
> Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/
> .__/\_,_/_/ /_/\_\ version 1.3.0 /_/ Using Scala version 2.10.4 (OpenJDK
> 64-Bit Server VM, Java 1.7.0_71) Type in expressions to have them
> evaluated. Type :help for more information.
> org.apache.spark.SparkException: Found both spark.executor.extraClassPath
> and SPARK_CLASSPATH. Use only the former. at
> org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$7.apply(SparkConf.scala:339)
> at
> org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$7.apply(SparkConf.scala:337)
> at scala.collection.immutable.List.foreach(List.scala:318) at
> org.apache.spark.SparkConf$$anonfun$validateSettings$6.apply(SparkConf.scala:337)
> at
> org.apache.spark.SparkConf$$anonfun$validateSettings$6.apply(SparkConf.scala:325)
> at scala.Option.foreach(Option.scala:236) at
> org.apache.spark.SparkConf.validateSettings(SparkConf.scala:325) at
> org.apache.spark.SparkContext.<init>(SparkContext.scala:197) at
> org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1016)
> at $iwC$$iwC.<init>(<console>:9) at $iwC.<init>(<console>:18) at
> <init>(<console>:20) at .<init>(<console>:24) at .<clinit>(<console>) at
> .<init>(<console>:7) at .<clinit>(<console>) at $print(<console>) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) at
> org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) at
> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856) at
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813) at
> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:123)
> at
> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:122)
> at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
> at
> org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:122)
> at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64) at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:973)
> at
> org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:157)
> at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64) at
> org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:106)
> at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:990)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
> at
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058) at
> org.apache.spark.repl.Main$.main(Main.scala:31) at
> org.apache.spark.repl.Main.main(Main.scala) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) at
> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> java.lang.NullPointerException at
> org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:141) at
> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:49) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at
> org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1027) at
> $iwC$$iwC.<init>(<console>:9)
> <console>:10: error: not found: value sqlContext import
> sqlContext.implicits._ ^ <console>:10: error: not found: value sqlContext
> import sqlContext.sql ^ scala> Stopping spark context. <console>:14: error:
> not found: value sc sc.stop() ^
>
> On Jul 22, 2015, at 8:36 PM, Alexander Bezzubov <ab...@nflabs.com>
> wrote:
>
> Hi,
>
> those are very valid steps indeed and you do not need to build custom
> spark version.
>
> Hope this helps!
>
> On Tue, Jul 21, 2015 at 7:46 PM, 江之源 <ji...@liulishuo.com> wrote:
>
> hi
> To install the zeppelin with z-manager is a helpless method, because i
> install the zeppelin in manually way and failed.I have tried many many
> times.
> My cluster is spark1.3.0 hadoop 2.0.0-cdh4.5.0,and the model is standalone.
> I will install the zeppelin manually right now so i wanna you check my
> steps:
>
> 1. git clone the repository from the github.
> 2. mvn clean package
> 3. mvn install -DskipTests -Dspark.version=1.3.0
> -Dhadoop.version=2.0.0-cdh4.5.0
> (did zeppelin support cdh4.5.0)
> Should i have to do the custom built spark
> like(-Dspark.version=1.1.0-Custom)
> 4.modify my master spark://...:7077
> Is it completed? or i lost something please tell me.
> thanks
> jzy
>
> 在 2015年7月21日，下午5:48，Alexander Bezzubov <ab...@nflabs.com> 写道：
>
> Hi,
>
> thank you for your interest in the project!
>
> It seems like the best way to get Zeppelin up and running in your case
> would be to build it manually with relevant Spark\Hadoop options as
> described here
> http://zeppelin.incubator.apache.org/docs/install/install.html
>
> Please, let me know if that helps.
>
> --
> BR,
> Alex
>
> On Tue, Jul 21, 2015 at 11:35 AM, 江之源 <ji...@liulishuo.com> wrote:
>
> hi
> i installed zeppelin some time before, but it always failed in my server
> cluster. i found the z-management Occasionally. I installed and success in
> my server. But when i wanna to read in my HDFS file like:
>
> sc.textFile("hdfs://llscluster/tmp/jzyresult/part-04093").count()
>
>
> it throw the errors in my cluster:Job aborted due to stage failure: Task 15
> in stage 6.0 failed 4 times, most recent failure: Lost task 15.3 in stage
> 6.0 (TID 386, lls7): java.io.EOFException
>
> when i modify it to the local model, it could read HDFS file successfully.
> My cluster is Spark1.3.0 Hadoop2.0.0-CDH4.5.0. but the install options just
> have Spark1.3.0 and Hadoop2.0.0-CDH-4.7.0. Is this the cause to read HDFS
> file failed?
> Look forward to your reply!
> THANK YOU!
> JZY
>
>
>
>
> --
> --
> Kind regards,
> Alexander.
>
>
>
>
>
> --
> --
> Kind regards,
> Alexander.
>
>
>
Re: a small question about zeppelin-manager
Posted by 江之源 <ji...@liulishuo.com>.
ok it works! 3q
And i have an another question about shell
The shell could work when type some simple commands.but it couldn’t work when i wanna to start a spark shell or spark submit even the sudo command. I wanna know the shell interpreter’s limit. why it always call
Process exited with an error: 1 (Exit value: 1) when i type sudo or spark-submit
and when i type spark-shell it feedback:
setting ulimit -m to 57671680
Spark assembly has been built with Hive, including Datanucleus jars on classpath
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/pipeline/zeppelin-manager/zeppelin-0.5.0-incubating-SNAPSHOT/interpreter/sh/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/pipeline/zeppelin-manager/zeppelin-0.5.0-incubating-SNAPSHOT/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/simple-spark.1.3.0/lib/spark-assembly-1.3.0-hadoop2.0.0-mr1-cdh4.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: (没有那个文件或目录)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685)
at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:78)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:1996)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:1996)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:1996)
at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:207)
at org.apache.spark.repl.SparkIMain.<init>(SparkIMain.scala:118)
at org.apache.spark.repl.SparkILoop$SparkILoopInterpreter.<init>(SparkILoop.scala:187)
at org.apache.spark.repl.SparkILoop.createInterpreter(SparkILoop.scala:216)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:948)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
log4j:ERROR Either File or DatePattern options are not set for appender [dailyfile].
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.3.0
/_/
Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_71)
Type in expressions to have them evaluated.
Type :help for more information.
org.apache.spark.SparkException: Found both spark.executor.extraClassPath and SPARK_CLASSPATH. Use only the former.
at org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$7.apply(SparkConf.scala:339)
at org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$7.apply(SparkConf.scala:337)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.SparkConf$$anonfun$validateSettings$6.apply(SparkConf.scala:337)
at org.apache.spark.SparkConf$$anonfun$validateSettings$6.apply(SparkConf.scala:325)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.SparkConf.validateSettings(SparkConf.scala:325)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:197)
at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1016)
at $iwC$$iwC.<init>(<console>:9)
at $iwC.<init>(<console>:18)
at <init>(<console>:20)
at .<init>(<console>:24)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:123)
at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:122)
at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:122)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:973)
at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:157)
at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:106)
at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:990)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
java.lang.NullPointerException
at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:141)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:49)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1027)
at $iwC$$iwC.<init>(<console>:9)
<console>:10: error: not found: value sqlContext
import sqlContext.implicits._
^
<console>:10: error: not found: value sqlContext
import sqlContext.sql
^
scala> Stopping spark context.
<console>:14: error: not found: value sc
sc.stop()
^
> On Jul 22, 2015, at 8:36 PM, Alexander Bezzubov <ab...@nflabs.com> wrote:
>
> Hi,
>
> those are very valid steps indeed and you do not need to build custom
> spark version.
>
> Hope this helps!
>
> On Tue, Jul 21, 2015 at 7:46 PM, 江之源 <ji...@liulishuo.com> wrote:
>> hi
>> To install the zeppelin with z-manager is a helpless method, because i
>> install the zeppelin in manually way and failed.I have tried many many
>> times.
>> My cluster is spark1.3.0 hadoop 2.0.0-cdh4.5.0,and the model is standalone.
>> I will install the zeppelin manually right now so i wanna you check my
>> steps:
>>
>> 1. git clone the repository from the github.
>> 2. mvn clean package
>> 3. mvn install -DskipTests -Dspark.version=1.3.0
>> -Dhadoop.version=2.0.0-cdh4.5.0
>> (did zeppelin support cdh4.5.0)
>> Should i have to do the custom built spark
>> like(-Dspark.version=1.1.0-Custom)
>> 4.modify my master spark://...:7077
>> Is it completed? or i lost something please tell me.
>> thanks
>> jzy
>>
>> 在 2015年7月21日，下午5:48，Alexander Bezzubov <ab...@nflabs.com> 写道：
>>
>> Hi,
>>
>> thank you for your interest in the project!
>>
>> It seems like the best way to get Zeppelin up and running in your case
>> would be to build it manually with relevant Spark\Hadoop options as
>> described here
>> http://zeppelin.incubator.apache.org/docs/install/install.html
>>
>> Please, let me know if that helps.
>>
>> --
>> BR,
>> Alex
>>
>> On Tue, Jul 21, 2015 at 11:35 AM, 江之源 <ji...@liulishuo.com> wrote:
>>
>> hi
>> i installed zeppelin some time before, but it always failed in my server
>> cluster. i found the z-management Occasionally. I installed and success in
>> my server. But when i wanna to read in my HDFS file like:
>>
>> sc.textFile("hdfs://llscluster/tmp/jzyresult/part-04093").count()
>>
>>
>> it throw the errors in my cluster:Job aborted due to stage failure: Task 15
>> in stage 6.0 failed 4 times, most recent failure: Lost task 15.3 in stage
>> 6.0 (TID 386, lls7): java.io.EOFException
>>
>> when i modify it to the local model, it could read HDFS file successfully.
>> My cluster is Spark1.3.0 Hadoop2.0.0-CDH4.5.0. but the install options just
>> have Spark1.3.0 and Hadoop2.0.0-CDH-4.7.0. Is this the cause to read HDFS
>> file failed?
>> Look forward to your reply!
>> THANK YOU!
>> JZY
>>
>>
>>
>>
>> --
>> --
>> Kind regards,
>> Alexander.
>>
>>
>
>
>
> --
> --
> Kind regards,
> Alexander.
Re: a small question about zeppelin-manager
Posted by Alexander Bezzubov <ab...@nflabs.com>.
Hi,
those are very valid steps indeed and you do not need to build custom
spark version.
Hope this helps!
On Tue, Jul 21, 2015 at 7:46 PM, 江之源 <ji...@liulishuo.com> wrote:
> hi
> To install the zeppelin with z-manager is a helpless method, because i
> install the zeppelin in manually way and failed.I have tried many many
> times.
> My cluster is spark1.3.0 hadoop 2.0.0-cdh4.5.0,and the model is standalone.
> I will install the zeppelin manually right now so i wanna you check my
> steps:
>
> 1. git clone the repository from the github.
> 2. mvn clean package
> 3. mvn install -DskipTests -Dspark.version=1.3.0
> -Dhadoop.version=2.0.0-cdh4.5.0
> (did zeppelin support cdh4.5.0)
> Should i have to do the custom built spark
> like(-Dspark.version=1.1.0-Custom)
> 4.modify my master spark://...:7077
> Is it completed? or i lost something please tell me.
> thanks
> jzy
>
> 在 2015年7月21日，下午5:48，Alexander Bezzubov <ab...@nflabs.com> 写道：
>
> Hi,
>
> thank you for your interest in the project!
>
> It seems like the best way to get Zeppelin up and running in your case
> would be to build it manually with relevant Spark\Hadoop options as
> described here
> http://zeppelin.incubator.apache.org/docs/install/install.html
>
> Please, let me know if that helps.
>
> --
> BR,
> Alex
>
> On Tue, Jul 21, 2015 at 11:35 AM, 江之源 <ji...@liulishuo.com> wrote:
>
> hi
> i installed zeppelin some time before, but it always failed in my server
> cluster. i found the z-management Occasionally. I installed and success in
> my server. But when i wanna to read in my HDFS file like:
>
> sc.textFile("hdfs://llscluster/tmp/jzyresult/part-04093").count()
>
>
> it throw the errors in my cluster:Job aborted due to stage failure: Task 15
> in stage 6.0 failed 4 times, most recent failure: Lost task 15.3 in stage
> 6.0 (TID 386, lls7): java.io.EOFException
>
> when i modify it to the local model, it could read HDFS file successfully.
> My cluster is Spark1.3.0 Hadoop2.0.0-CDH4.5.0. but the install options just
> have Spark1.3.0 and Hadoop2.0.0-CDH-4.7.0. Is this the cause to read HDFS
> file failed?
> Look forward to your reply!
> THANK YOU!
> JZY
>
>
>
>
> --
> --
> Kind regards,
> Alexander.
>
>
--
--
Kind regards,
Alexander.
Re: a small question about zeppelin-manager
Posted by 江之源 <ji...@liulishuo.com>.
hi
To install the zeppelin with z-manager is a helpless method, because i install the zeppelin in manually way and failed.I have tried many many times.
My cluster is spark1.3.0 hadoop 2.0.0-cdh4.5.0,and the model is standalone.
I will install the zeppelin manually right now so i wanna you check my steps:
1. git clone the repository from the github.
2. mvn clean package
3. mvn install -DskipTests -Dspark.version=1.3.0 -Dhadoop.version=2.0.0-cdh4.5.0
(did zeppelin support cdh4.5.0)
Should i have to do the custom built spark like(-Dspark.version=1.1.0-Custom)
4.modify my master spark://...:7077 <spark://...:7077>
Is it completed? or i lost something please tell me.
thanks
jzy
> 在 2015年7月21日，下午5:48，Alexander Bezzubov <ab...@nflabs.com> 写道：
>
> Hi,
>
> thank you for your interest in the project!
>
> It seems like the best way to get Zeppelin up and running in your case
> would be to build it manually with relevant Spark\Hadoop options as
> described here http://zeppelin.incubator.apache.org/docs/install/install.html
>
> Please, let me know if that helps.
>
> --
> BR,
> Alex
>
> On Tue, Jul 21, 2015 at 11:35 AM, 江之源 <ji...@liulishuo.com> wrote:
>> hi
>> i installed zeppelin some time before, but it always failed in my server
>> cluster. i found the z-management Occasionally. I installed and success in
>> my server. But when i wanna to read in my HDFS file like:
>>
>> sc.textFile("hdfs://llscluster/tmp/jzyresult/part-04093").count()
>>
>>
>> it throw the errors in my cluster:Job aborted due to stage failure: Task 15
>> in stage 6.0 failed 4 times, most recent failure: Lost task 15.3 in stage
>> 6.0 (TID 386, lls7): java.io.EOFException
>>
>> when i modify it to the local model, it could read HDFS file successfully.
>> My cluster is Spark1.3.0 Hadoop2.0.0-CDH4.5.0. but the install options just
>> have Spark1.3.0 and Hadoop2.0.0-CDH-4.7.0. Is this the cause to read HDFS
>> file failed?
>> Look forward to your reply!
>> THANK YOU!
>> JZY
>
>
>
> --
> --
> Kind regards,
> Alexander.
Re: a small question about zeppelin-manager
Posted by Alexander Bezzubov <ab...@nflabs.com>.
Hi,
thank you for your interest in the project!
It seems like the best way to get Zeppelin up and running in your case
would be to build it manually with relevant Spark\Hadoop options as
described here http://zeppelin.incubator.apache.org/docs/install/install.html
Please, let me know if that helps.
--
BR,
Alex
On Tue, Jul 21, 2015 at 11:35 AM, 江之源 <ji...@liulishuo.com> wrote:
> hi
> i installed zeppelin some time before, but it always failed in my server
> cluster. i found the z-management Occasionally. I installed and success in
> my server. But when i wanna to read in my HDFS file like:
>
> sc.textFile("hdfs://llscluster/tmp/jzyresult/part-04093").count()
>
>
> it throw the errors in my cluster:Job aborted due to stage failure: Task 15
> in stage 6.0 failed 4 times, most recent failure: Lost task 15.3 in stage
> 6.0 (TID 386, lls7): java.io.EOFException
>
> when i modify it to the local model, it could read HDFS file successfully.
> My cluster is Spark1.3.0 Hadoop2.0.0-CDH4.5.0. but the install options just
> have Spark1.3.0 and Hadoop2.0.0-CDH-4.7.0. Is this the cause to read HDFS
> file failed?
> Look forward to your reply!
> THANK YOU!
> JZY
--
--
Kind regards,
Alexander.