You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by scwf <wa...@huawei.com> on 2014/09/02 10:39:00 UTC
about spark assembly jar
hi, all
I suggest spark not use assembly jar as default run-time dependency(spark-submit/spark-class depend on assembly jar),use a library of all 3rd dependency jar like hadoop/hive/hbase more reasonable.
1 assembly jar packaged all 3rd jars into a big one, so we need rebuild this jar if we want to update the version of some component(such as hadoop)
2 in our practice with spark, sometimes we meet jar compatibility issue, it is hard to diagnose compatibility issue with assembly jar
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: about spark assembly jar
Posted by scwf <wa...@huawei.com>.
Yea, SSD + SPARK_PREPEND_CLASSES is great for iterative development!
Then why it is ok with a bag of 3rd jars but throw error with assembly jar, any one have idea?
On 2014/9/3 2:57, Cheng Lian wrote:
> Cool, didn't notice that, thanks Josh!
>
>
> On Tue, Sep 2, 2014 at 11:55 AM, Josh Rosen <rosenville@gmail.com <ma...@gmail.com>> wrote:
>
> SPARK_PREPEND_CLASSES is documented on the Spark Wiki (which could probably be easier to find): https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools
>
>
> On September 2, 2014 at 11:53:49 AM, Cheng Lian (lian.cs.zju@gmail.com <ma...@gmail.com>) wrote:
>
>> Yea, SSD + SPARK_PREPEND_CLASSES totally changed my life :)
>>
>> Maybe we should add a "developer notes" page to document all these useful
>> black magic.
>>
>>
>> On Tue, Sep 2, 2014 at 10:54 AM, Reynold Xin <rxin@databricks.com <ma...@databricks.com>> wrote:
>>
>> > Having a SSD help tremendously with assembly time.
>> >
>> > Without that, you can do the following in order for Spark to pick up the
>> > compiled classes before assembly at runtime.
>> >
>> > export SPARK_PREPEND_CLASSES=true
>> >
>> >
>> > On Tue, Sep 2, 2014 at 9:10 AM, Sandy Ryza <sandy.ryza@cloudera.com <ma...@cloudera.com>>
>> > wrote:
>> >
>> > > This doesn't help for every dependency, but Spark provides an option to
>> > > build the assembly jar without Hadoop and its dependencies. We make use
>> > of
>> > > this in CDH packaging.
>> > >
>> > > -Sandy
>> > >
>> > >
>> > > On Tue, Sep 2, 2014 at 2:12 AM, scwf <wangfei1@huawei.com <ma...@huawei.com>> wrote:
>> > >
>> > > > Hi sean owen,
>> > > > here are some problems when i used assembly jar
>> > > > 1 i put spark-assembly-*.jar to the lib directory of my application, it
>> > > > throw compile error
>> > > >
>> > > > Error:scalac: Error: class scala.reflect.BeanInfo not found.
>> > > > scala.tools.nsc.MissingRequirementError: class scala.reflect.BeanInfo
>> > not
>> > > > found.
>> > > >
>> > > > at scala.tools.nsc.symtab.Definitions$definitions$.
>> > > > getModuleOrClass(Definitions.scala:655)
>> > > >
>> > > > at scala.tools.nsc.symtab.Definitions$definitions$.
>> > > > getClass(Definitions.scala:608)
>> > > >
>> > > > at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator.<
>> > > > init>(GenJVM.scala:127)
>> > > >
>> > > > at scala.tools.nsc.backend.jvm.GenJVM$JvmPhase.run(GenJVM.
>> > > > scala:85)
>> > > >
>> > > > at scala.tools.nsc.Global$Run.compileSources(Global.scala:953)
>> > > >
>> > > > at scala.tools.nsc.Global$Run.compile(Global.scala:1041)
>> > > >
>> > > > at xsbt.CachedCompiler0.run(CompilerInterface.scala:126)
>> > > >
>> > > > at
>> > > xsbt.CachedCompiler0.liftedTree1$1(CompilerInterface.scala:102)
>> > > >
>> > > > at xsbt.CachedCompiler0.run(CompilerInterface.scala:102)
>> > > >
>> > > > at xsbt.CompilerInterface.run(CompilerInterface.scala:27)
>> > > >
>> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > > >
>> > > > at sun.reflect.NativeMethodAccessorImpl.invoke(
>> > > > NativeMethodAccessorImpl.java:39)
>> > > >
>> > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> > > > DelegatingMethodAccessorImpl.java:25)
>> > > >
>> > > > at java.lang.reflect.Method.invoke(Method.java:597)
>> > > >
>> > > > at sbt.compiler.AnalyzingCompiler.call(
>> > > > AnalyzingCompiler.scala:102)
>> > > >
>> > > > at sbt.compiler.AnalyzingCompiler.compile(
>> > > > AnalyzingCompiler.scala:48)
>> > > >
>> > > > at sbt.compiler.AnalyzingCompiler.compile(
>> > > > AnalyzingCompiler.scala:41)
>> > > >
>> > > > at org.jetbrains.jps.incremental.scala.local.
>> > > > IdeaIncrementalCompiler.compile(IdeaIncrementalCompiler.scala:28)
>> > > >
>> > > > at org.jetbrains.jps.incremental.scala.local.LocalServer.
>> > > > compile(LocalServer.scala:25)
>> > > >
>> > > > at org.jetbrains.jps.incremental.scala.remote.Main$.make(Main.
>> > > > scala:58)
>> > > >
>> > > > at org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(
>> > > > Main.scala:21)
>> > > >
>> > > > at org.jetbrains.jps.incremental.scala.remote.Main.nailMain(
>> > > > Main.scala)
>> > > >
>> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > > >
>> > > > at sun.reflect.NativeMethodAccessorImpl.invoke(
>> > > > NativeMethodAccessorImpl.java:39)
>> > > >
>> > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> > > > DelegatingMethodAccessorImpl.java:25)
>> > > >
>> > > > at java.lang.reflect.Method.invoke(Method.java:597)
>> > > >
>> > > > at
>> > com.martiansoftware.nailgun.NGSession.run(NGSession.java:319)
>> > > > 2 i test my branch which updated hive version to org.apache.hive 0.13.1
>> > > > it run successfully when use a bag of 3rd jars as dependency but
>> > throw
>> > > > error using assembly jar, it seems assembly jar lead to conflict
>> > > > ERROR DDLTask: java.lang.NoSuchFieldError: doubleTypeInfo
>> > > > at org.apache.hadoop.hive.ql.io.parquet.serde.
>> > > > ArrayWritableObjectInspector.getObjectInspector(
>> > > > ArrayWritableObjectInspector.java:66)
>> > > > at org.apache.hadoop.hive.ql.io.parquet.serde.
>> > > >
>> > ArrayWritableObjectInspector.<init>(ArrayWritableObjectInspector.java:59)
>> > > > at org.apache.hadoop.hive.ql.io.parquet.serde.
>> > > > ParquetHiveSerDe.initialize(ParquetHiveSerDe.java:113)
>> > > > at org.apache.hadoop.hive.metastore.MetaStoreUtils.
>> > > > getDeserializer(MetaStoreUtils.java:339)
>> > > > at org.apache.hadoop.hive.ql.metadata.Table.
>> > > > getDeserializerFromMetaStore(Table.java:283)
>> > > > at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(
>> > > > Table.java:189)
>> > > > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(
>> > > > Hive.java:597)
>> > > > at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(
>> > > > DDLTask.java:4194)
>> > > > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.
>> > > > java:281)
>> > > > at
>> > org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
>> > > > at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(
>> > > > TaskRunner.java:85)
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On 2014/9/2 16:45, Sean Owen wrote:
>> > > >
>> > > >> Hm, are you suggesting that the Spark distribution be a bag of 100
>> > > >> JARs? It doesn't quite seem reasonable. It does not remove version
>> > > >> conflicts, just pushes them to run-time, which isn't good. The
>> > > >> assembly is also necessary because that's where shading happens. In
>> > > >> development, you want to run against exactly what will be used in a
>> > > >> real Spark distro.
>> > > >>
>> > > >> On Tue, Sep 2, 2014 at 9:39 AM, scwf <wangfei1@huawei.com <ma...@huawei.com>> wrote:
>> > > >>
>> > > >>> hi, all
>> > > >>> I suggest spark not use assembly jar as default run-time
>> > > >>> dependency(spark-submit/spark-class depend on assembly jar),use a
>> > > >>> library of
>> > > >>> all 3rd dependency jar like hadoop/hive/hbase more reasonable.
>> > > >>>
>> > > >>> 1 assembly jar packaged all 3rd jars into a big one, so we need
>> > > >>> rebuild
>> > > >>> this jar if we want to update the version of some component(such as
>> > > >>> hadoop)
>> > > >>> 2 in our practice with spark, sometimes we meet jar compatibility
>> > > >>> issue,
>> > > >>> it is hard to diagnose compatibility issue with assembly jar
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>> ---------------------------------------------------------------------
>> > > >>> To unsubscribe, e-mail:dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>> > > >>> For additional commands, e-mail:dev-help@spark.apache.org <ma...@spark.apache.org>
>> > > >>>
>> > > >>>
>> > > >>
>> > > >>
>> > > >
>> > > >
>> > > > ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail:dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>> > > > For additional commands, e-mail:dev-help@spark.apache.org <ma...@spark.apache.org>
>> > > >
>> > > >
>> > >
>> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: about spark assembly jar
Posted by Cheng Lian <li...@gmail.com>.
Cool, didn't notice that, thanks Josh!
On Tue, Sep 2, 2014 at 11:55 AM, Josh Rosen <ro...@gmail.com> wrote:
> SPARK_PREPEND_CLASSES is documented on the Spark Wiki (which could
> probably be easier to find):
> https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools
>
>
> On September 2, 2014 at 11:53:49 AM, Cheng Lian (lian.cs.zju@gmail.com)
> wrote:
>
> Yea, SSD + SPARK_PREPEND_CLASSES totally changed my life :)
>
> Maybe we should add a "developer notes" page to document all these useful
> black magic.
>
>
> On Tue, Sep 2, 2014 at 10:54 AM, Reynold Xin <rx...@databricks.com> wrote:
>
> > Having a SSD help tremendously with assembly time.
> >
> > Without that, you can do the following in order for Spark to pick up the
> > compiled classes before assembly at runtime.
> >
> > export SPARK_PREPEND_CLASSES=true
> >
> >
> > On Tue, Sep 2, 2014 at 9:10 AM, Sandy Ryza <sa...@cloudera.com>
> > wrote:
> >
> > > This doesn't help for every dependency, but Spark provides an option
> to
> > > build the assembly jar without Hadoop and its dependencies. We make
> use
> > of
> > > this in CDH packaging.
> > >
> > > -Sandy
> > >
> > >
> > > On Tue, Sep 2, 2014 at 2:12 AM, scwf <wa...@huawei.com> wrote:
> > >
> > > > Hi sean owen,
> > > > here are some problems when i used assembly jar
> > > > 1 i put spark-assembly-*.jar to the lib directory of my application,
> it
> > > > throw compile error
> > > >
> > > > Error:scalac: Error: class scala.reflect.BeanInfo not found.
> > > > scala.tools.nsc.MissingRequirementError: class
> scala.reflect.BeanInfo
> > not
> > > > found.
> > > >
> > > > at scala.tools.nsc.symtab.Definitions$definitions$.
> > > > getModuleOrClass(Definitions.scala:655)
> > > >
> > > > at scala.tools.nsc.symtab.Definitions$definitions$.
> > > > getClass(Definitions.scala:608)
> > > >
> > > > at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator.<
> > > > init>(GenJVM.scala:127)
> > > >
> > > > at scala.tools.nsc.backend.jvm.GenJVM$JvmPhase.run(GenJVM.
> > > > scala:85)
> > > >
> > > > at scala.tools.nsc.Global$Run.compileSources(Global.scala:953)
> > > >
> > > > at scala.tools.nsc.Global$Run.compile(Global.scala:1041)
> > > >
> > > > at xsbt.CachedCompiler0.run(CompilerInterface.scala:126)
> > > >
> > > > at
> > > xsbt.CachedCompiler0.liftedTree1$1(CompilerInterface.scala:102)
> > > >
> > > > at xsbt.CachedCompiler0.run(CompilerInterface.scala:102)
> > > >
> > > > at xsbt.CompilerInterface.run(CompilerInterface.scala:27)
> > > >
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > >
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > > > NativeMethodAccessorImpl.java:39)
> > > >
> > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > > DelegatingMethodAccessorImpl.java:25)
> > > >
> > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > >
> > > > at sbt.compiler.AnalyzingCompiler.call(
> > > > AnalyzingCompiler.scala:102)
> > > >
> > > > at sbt.compiler.AnalyzingCompiler.compile(
> > > > AnalyzingCompiler.scala:48)
> > > >
> > > > at sbt.compiler.AnalyzingCompiler.compile(
> > > > AnalyzingCompiler.scala:41)
> > > >
> > > > at org.jetbrains.jps.incremental.scala.local.
> > > > IdeaIncrementalCompiler.compile(IdeaIncrementalCompiler.scala:28)
> > > >
> > > > at org.jetbrains.jps.incremental.scala.local.LocalServer.
> > > > compile(LocalServer.scala:25)
> > > >
> > > > at org.jetbrains.jps.incremental.scala.remote.Main$.make(Main.
> > > > scala:58)
> > > >
> > > > at org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(
> > > > Main.scala:21)
> > > >
> > > > at org.jetbrains.jps.incremental.scala.remote.Main.nailMain(
> > > > Main.scala)
> > > >
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > >
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > > > NativeMethodAccessorImpl.java:39)
> > > >
> > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > > DelegatingMethodAccessorImpl.java:25)
> > > >
> > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > >
> > > > at
> > com.martiansoftware.nailgun.NGSession.run(NGSession.java:319)
> > > > 2 i test my branch which updated hive version to org.apache.hive
> 0.13.1
> > > > it run successfully when use a bag of 3rd jars as dependency but
> > throw
> > > > error using assembly jar, it seems assembly jar lead to conflict
> > > > ERROR DDLTask: java.lang.NoSuchFieldError: doubleTypeInfo
> > > > at org.apache.hadoop.hive.ql.io.parquet.serde.
> > > > ArrayWritableObjectInspector.getObjectInspector(
> > > > ArrayWritableObjectInspector.java:66)
> > > > at org.apache.hadoop.hive.ql.io.parquet.serde.
> > > >
> >
> ArrayWritableObjectInspector.<init>(ArrayWritableObjectInspector.java:59)
> > > > at org.apache.hadoop.hive.ql.io.parquet.serde.
> > > > ParquetHiveSerDe.initialize(ParquetHiveSerDe.java:113)
> > > > at org.apache.hadoop.hive.metastore.MetaStoreUtils.
> > > > getDeserializer(MetaStoreUtils.java:339)
> > > > at org.apache.hadoop.hive.ql.metadata.Table.
> > > > getDeserializerFromMetaStore(Table.java:283)
> > > > at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(
> > > > Table.java:189)
> > > > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(
> > > > Hive.java:597)
> > > > at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(
> > > > DDLTask.java:4194)
> > > > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.
> > > > java:281)
> > > > at
> > org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> > > > at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(
> > > > TaskRunner.java:85)
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 2014/9/2 16:45, Sean Owen wrote:
> > > >
> > > >> Hm, are you suggesting that the Spark distribution be a bag of 100
> > > >> JARs? It doesn't quite seem reasonable. It does not remove version
> > > >> conflicts, just pushes them to run-time, which isn't good. The
> > > >> assembly is also necessary because that's where shading happens. In
> > > >> development, you want to run against exactly what will be used in a
> > > >> real Spark distro.
> > > >>
> > > >> On Tue, Sep 2, 2014 at 9:39 AM, scwf <wa...@huawei.com> wrote:
> > > >>
> > > >>> hi, all
> > > >>> I suggest spark not use assembly jar as default run-time
> > > >>> dependency(spark-submit/spark-class depend on assembly jar),use a
> > > >>> library of
> > > >>> all 3rd dependency jar like hadoop/hive/hbase more reasonable.
> > > >>>
> > > >>> 1 assembly jar packaged all 3rd jars into a big one, so we need
> > > >>> rebuild
> > > >>> this jar if we want to update the version of some component(such
> as
> > > >>> hadoop)
> > > >>> 2 in our practice with spark, sometimes we meet jar compatibility
> > > >>> issue,
> > > >>> it is hard to diagnose compatibility issue with assembly jar
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> ---------------------------------------------------------------------
> > > >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > > >>> For additional commands, e-mail: dev-help@spark.apache.org
> > > >>>
> > > >>>
> > > >>
> > > >>
> > > >
> > > >
> > > >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > > > For additional commands, e-mail: dev-help@spark.apache.org
> > > >
> > > >
> > >
> >
>
>
Re: about spark assembly jar
Posted by Josh Rosen <ro...@gmail.com>.
SPARK_PREPEND_CLASSES is documented on the Spark Wiki (which could probably be easier to find): https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools
On September 2, 2014 at 11:53:49 AM, Cheng Lian (lian.cs.zju@gmail.com) wrote:
Yea, SSD + SPARK_PREPEND_CLASSES totally changed my life :)
Maybe we should add a "developer notes" page to document all these useful
black magic.
On Tue, Sep 2, 2014 at 10:54 AM, Reynold Xin <rx...@databricks.com> wrote:
> Having a SSD help tremendously with assembly time.
>
> Without that, you can do the following in order for Spark to pick up the
> compiled classes before assembly at runtime.
>
> export SPARK_PREPEND_CLASSES=true
>
>
> On Tue, Sep 2, 2014 at 9:10 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> > This doesn't help for every dependency, but Spark provides an option to
> > build the assembly jar without Hadoop and its dependencies. We make use
> of
> > this in CDH packaging.
> >
> > -Sandy
> >
> >
> > On Tue, Sep 2, 2014 at 2:12 AM, scwf <wa...@huawei.com> wrote:
> >
> > > Hi sean owen,
> > > here are some problems when i used assembly jar
> > > 1 i put spark-assembly-*.jar to the lib directory of my application, it
> > > throw compile error
> > >
> > > Error:scalac: Error: class scala.reflect.BeanInfo not found.
> > > scala.tools.nsc.MissingRequirementError: class scala.reflect.BeanInfo
> not
> > > found.
> > >
> > > at scala.tools.nsc.symtab.Definitions$definitions$.
> > > getModuleOrClass(Definitions.scala:655)
> > >
> > > at scala.tools.nsc.symtab.Definitions$definitions$.
> > > getClass(Definitions.scala:608)
> > >
> > > at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator.<
> > > init>(GenJVM.scala:127)
> > >
> > > at scala.tools.nsc.backend.jvm.GenJVM$JvmPhase.run(GenJVM.
> > > scala:85)
> > >
> > > at scala.tools.nsc.Global$Run.compileSources(Global.scala:953)
> > >
> > > at scala.tools.nsc.Global$Run.compile(Global.scala:1041)
> > >
> > > at xsbt.CachedCompiler0.run(CompilerInterface.scala:126)
> > >
> > > at
> > xsbt.CachedCompiler0.liftedTree1$1(CompilerInterface.scala:102)
> > >
> > > at xsbt.CachedCompiler0.run(CompilerInterface.scala:102)
> > >
> > > at xsbt.CompilerInterface.run(CompilerInterface.scala:27)
> > >
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >
> > > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > > NativeMethodAccessorImpl.java:39)
> > >
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:25)
> > >
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > >
> > > at sbt.compiler.AnalyzingCompiler.call(
> > > AnalyzingCompiler.scala:102)
> > >
> > > at sbt.compiler.AnalyzingCompiler.compile(
> > > AnalyzingCompiler.scala:48)
> > >
> > > at sbt.compiler.AnalyzingCompiler.compile(
> > > AnalyzingCompiler.scala:41)
> > >
> > > at org.jetbrains.jps.incremental.scala.local.
> > > IdeaIncrementalCompiler.compile(IdeaIncrementalCompiler.scala:28)
> > >
> > > at org.jetbrains.jps.incremental.scala.local.LocalServer.
> > > compile(LocalServer.scala:25)
> > >
> > > at org.jetbrains.jps.incremental.scala.remote.Main$.make(Main.
> > > scala:58)
> > >
> > > at org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(
> > > Main.scala:21)
> > >
> > > at org.jetbrains.jps.incremental.scala.remote.Main.nailMain(
> > > Main.scala)
> > >
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >
> > > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > > NativeMethodAccessorImpl.java:39)
> > >
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:25)
> > >
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > >
> > > at
> com.martiansoftware.nailgun.NGSession.run(NGSession.java:319)
> > > 2 i test my branch which updated hive version to org.apache.hive 0.13.1
> > > it run successfully when use a bag of 3rd jars as dependency but
> throw
> > > error using assembly jar, it seems assembly jar lead to conflict
> > > ERROR DDLTask: java.lang.NoSuchFieldError: doubleTypeInfo
> > > at org.apache.hadoop.hive.ql.io.parquet.serde.
> > > ArrayWritableObjectInspector.getObjectInspector(
> > > ArrayWritableObjectInspector.java:66)
> > > at org.apache.hadoop.hive.ql.io.parquet.serde.
> > >
> ArrayWritableObjectInspector.<init>(ArrayWritableObjectInspector.java:59)
> > > at org.apache.hadoop.hive.ql.io.parquet.serde.
> > > ParquetHiveSerDe.initialize(ParquetHiveSerDe.java:113)
> > > at org.apache.hadoop.hive.metastore.MetaStoreUtils.
> > > getDeserializer(MetaStoreUtils.java:339)
> > > at org.apache.hadoop.hive.ql.metadata.Table.
> > > getDeserializerFromMetaStore(Table.java:283)
> > > at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(
> > > Table.java:189)
> > > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(
> > > Hive.java:597)
> > > at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(
> > > DDLTask.java:4194)
> > > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.
> > > java:281)
> > > at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> > > at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(
> > > TaskRunner.java:85)
> > >
> > >
> > >
> > >
> > >
> > > On 2014/9/2 16:45, Sean Owen wrote:
> > >
> > >> Hm, are you suggesting that the Spark distribution be a bag of 100
> > >> JARs? It doesn't quite seem reasonable. It does not remove version
> > >> conflicts, just pushes them to run-time, which isn't good. The
> > >> assembly is also necessary because that's where shading happens. In
> > >> development, you want to run against exactly what will be used in a
> > >> real Spark distro.
> > >>
> > >> On Tue, Sep 2, 2014 at 9:39 AM, scwf <wa...@huawei.com> wrote:
> > >>
> > >>> hi, all
> > >>> I suggest spark not use assembly jar as default run-time
> > >>> dependency(spark-submit/spark-class depend on assembly jar),use a
> > >>> library of
> > >>> all 3rd dependency jar like hadoop/hive/hbase more reasonable.
> > >>>
> > >>> 1 assembly jar packaged all 3rd jars into a big one, so we need
> > >>> rebuild
> > >>> this jar if we want to update the version of some component(such as
> > >>> hadoop)
> > >>> 2 in our practice with spark, sometimes we meet jar compatibility
> > >>> issue,
> > >>> it is hard to diagnose compatibility issue with assembly jar
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > >>> For additional commands, e-mail: dev-help@spark.apache.org
> > >>>
> > >>>
> > >>
> > >>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > > For additional commands, e-mail: dev-help@spark.apache.org
> > >
> > >
> >
>
Re: about spark assembly jar
Posted by Cheng Lian <li...@gmail.com>.
Yea, SSD + SPARK_PREPEND_CLASSES totally changed my life :)
Maybe we should add a "developer notes" page to document all these useful
black magic.
On Tue, Sep 2, 2014 at 10:54 AM, Reynold Xin <rx...@databricks.com> wrote:
> Having a SSD help tremendously with assembly time.
>
> Without that, you can do the following in order for Spark to pick up the
> compiled classes before assembly at runtime.
>
> export SPARK_PREPEND_CLASSES=true
>
>
> On Tue, Sep 2, 2014 at 9:10 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> > This doesn't help for every dependency, but Spark provides an option to
> > build the assembly jar without Hadoop and its dependencies. We make use
> of
> > this in CDH packaging.
> >
> > -Sandy
> >
> >
> > On Tue, Sep 2, 2014 at 2:12 AM, scwf <wa...@huawei.com> wrote:
> >
> > > Hi sean owen,
> > > here are some problems when i used assembly jar
> > > 1 i put spark-assembly-*.jar to the lib directory of my application, it
> > > throw compile error
> > >
> > > Error:scalac: Error: class scala.reflect.BeanInfo not found.
> > > scala.tools.nsc.MissingRequirementError: class scala.reflect.BeanInfo
> not
> > > found.
> > >
> > > at scala.tools.nsc.symtab.Definitions$definitions$.
> > > getModuleOrClass(Definitions.scala:655)
> > >
> > > at scala.tools.nsc.symtab.Definitions$definitions$.
> > > getClass(Definitions.scala:608)
> > >
> > > at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator.<
> > > init>(GenJVM.scala:127)
> > >
> > > at scala.tools.nsc.backend.jvm.GenJVM$JvmPhase.run(GenJVM.
> > > scala:85)
> > >
> > > at scala.tools.nsc.Global$Run.compileSources(Global.scala:953)
> > >
> > > at scala.tools.nsc.Global$Run.compile(Global.scala:1041)
> > >
> > > at xsbt.CachedCompiler0.run(CompilerInterface.scala:126)
> > >
> > > at
> > xsbt.CachedCompiler0.liftedTree1$1(CompilerInterface.scala:102)
> > >
> > > at xsbt.CachedCompiler0.run(CompilerInterface.scala:102)
> > >
> > > at xsbt.CompilerInterface.run(CompilerInterface.scala:27)
> > >
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >
> > > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > > NativeMethodAccessorImpl.java:39)
> > >
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:25)
> > >
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > >
> > > at sbt.compiler.AnalyzingCompiler.call(
> > > AnalyzingCompiler.scala:102)
> > >
> > > at sbt.compiler.AnalyzingCompiler.compile(
> > > AnalyzingCompiler.scala:48)
> > >
> > > at sbt.compiler.AnalyzingCompiler.compile(
> > > AnalyzingCompiler.scala:41)
> > >
> > > at org.jetbrains.jps.incremental.scala.local.
> > > IdeaIncrementalCompiler.compile(IdeaIncrementalCompiler.scala:28)
> > >
> > > at org.jetbrains.jps.incremental.scala.local.LocalServer.
> > > compile(LocalServer.scala:25)
> > >
> > > at org.jetbrains.jps.incremental.scala.remote.Main$.make(Main.
> > > scala:58)
> > >
> > > at org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(
> > > Main.scala:21)
> > >
> > > at org.jetbrains.jps.incremental.scala.remote.Main.nailMain(
> > > Main.scala)
> > >
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >
> > > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > > NativeMethodAccessorImpl.java:39)
> > >
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:25)
> > >
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > >
> > > at
> com.martiansoftware.nailgun.NGSession.run(NGSession.java:319)
> > > 2 i test my branch which updated hive version to org.apache.hive 0.13.1
> > > it run successfully when use a bag of 3rd jars as dependency but
> throw
> > > error using assembly jar, it seems assembly jar lead to conflict
> > > ERROR DDLTask: java.lang.NoSuchFieldError: doubleTypeInfo
> > > at org.apache.hadoop.hive.ql.io.parquet.serde.
> > > ArrayWritableObjectInspector.getObjectInspector(
> > > ArrayWritableObjectInspector.java:66)
> > > at org.apache.hadoop.hive.ql.io.parquet.serde.
> > >
> ArrayWritableObjectInspector.<init>(ArrayWritableObjectInspector.java:59)
> > > at org.apache.hadoop.hive.ql.io.parquet.serde.
> > > ParquetHiveSerDe.initialize(ParquetHiveSerDe.java:113)
> > > at org.apache.hadoop.hive.metastore.MetaStoreUtils.
> > > getDeserializer(MetaStoreUtils.java:339)
> > > at org.apache.hadoop.hive.ql.metadata.Table.
> > > getDeserializerFromMetaStore(Table.java:283)
> > > at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(
> > > Table.java:189)
> > > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(
> > > Hive.java:597)
> > > at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(
> > > DDLTask.java:4194)
> > > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.
> > > java:281)
> > > at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> > > at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(
> > > TaskRunner.java:85)
> > >
> > >
> > >
> > >
> > >
> > > On 2014/9/2 16:45, Sean Owen wrote:
> > >
> > >> Hm, are you suggesting that the Spark distribution be a bag of 100
> > >> JARs? It doesn't quite seem reasonable. It does not remove version
> > >> conflicts, just pushes them to run-time, which isn't good. The
> > >> assembly is also necessary because that's where shading happens. In
> > >> development, you want to run against exactly what will be used in a
> > >> real Spark distro.
> > >>
> > >> On Tue, Sep 2, 2014 at 9:39 AM, scwf <wa...@huawei.com> wrote:
> > >>
> > >>> hi, all
> > >>> I suggest spark not use assembly jar as default run-time
> > >>> dependency(spark-submit/spark-class depend on assembly jar),use a
> > >>> library of
> > >>> all 3rd dependency jar like hadoop/hive/hbase more reasonable.
> > >>>
> > >>> 1 assembly jar packaged all 3rd jars into a big one, so we need
> > >>> rebuild
> > >>> this jar if we want to update the version of some component(such as
> > >>> hadoop)
> > >>> 2 in our practice with spark, sometimes we meet jar compatibility
> > >>> issue,
> > >>> it is hard to diagnose compatibility issue with assembly jar
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > >>> For additional commands, e-mail: dev-help@spark.apache.org
> > >>>
> > >>>
> > >>
> > >>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > > For additional commands, e-mail: dev-help@spark.apache.org
> > >
> > >
> >
>
Re: about spark assembly jar
Posted by Reynold Xin <rx...@databricks.com>.
Having a SSD help tremendously with assembly time.
Without that, you can do the following in order for Spark to pick up the
compiled classes before assembly at runtime.
export SPARK_PREPEND_CLASSES=true
On Tue, Sep 2, 2014 at 9:10 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> This doesn't help for every dependency, but Spark provides an option to
> build the assembly jar without Hadoop and its dependencies. We make use of
> this in CDH packaging.
>
> -Sandy
>
>
> On Tue, Sep 2, 2014 at 2:12 AM, scwf <wa...@huawei.com> wrote:
>
> > Hi sean owen,
> > here are some problems when i used assembly jar
> > 1 i put spark-assembly-*.jar to the lib directory of my application, it
> > throw compile error
> >
> > Error:scalac: Error: class scala.reflect.BeanInfo not found.
> > scala.tools.nsc.MissingRequirementError: class scala.reflect.BeanInfo not
> > found.
> >
> > at scala.tools.nsc.symtab.Definitions$definitions$.
> > getModuleOrClass(Definitions.scala:655)
> >
> > at scala.tools.nsc.symtab.Definitions$definitions$.
> > getClass(Definitions.scala:608)
> >
> > at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator.<
> > init>(GenJVM.scala:127)
> >
> > at scala.tools.nsc.backend.jvm.GenJVM$JvmPhase.run(GenJVM.
> > scala:85)
> >
> > at scala.tools.nsc.Global$Run.compileSources(Global.scala:953)
> >
> > at scala.tools.nsc.Global$Run.compile(Global.scala:1041)
> >
> > at xsbt.CachedCompiler0.run(CompilerInterface.scala:126)
> >
> > at
> xsbt.CachedCompiler0.liftedTree1$1(CompilerInterface.scala:102)
> >
> > at xsbt.CachedCompiler0.run(CompilerInterface.scala:102)
> >
> > at xsbt.CompilerInterface.run(CompilerInterface.scala:27)
> >
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >
> > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:39)
> >
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:25)
> >
> > at java.lang.reflect.Method.invoke(Method.java:597)
> >
> > at sbt.compiler.AnalyzingCompiler.call(
> > AnalyzingCompiler.scala:102)
> >
> > at sbt.compiler.AnalyzingCompiler.compile(
> > AnalyzingCompiler.scala:48)
> >
> > at sbt.compiler.AnalyzingCompiler.compile(
> > AnalyzingCompiler.scala:41)
> >
> > at org.jetbrains.jps.incremental.scala.local.
> > IdeaIncrementalCompiler.compile(IdeaIncrementalCompiler.scala:28)
> >
> > at org.jetbrains.jps.incremental.scala.local.LocalServer.
> > compile(LocalServer.scala:25)
> >
> > at org.jetbrains.jps.incremental.scala.remote.Main$.make(Main.
> > scala:58)
> >
> > at org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(
> > Main.scala:21)
> >
> > at org.jetbrains.jps.incremental.scala.remote.Main.nailMain(
> > Main.scala)
> >
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >
> > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:39)
> >
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:25)
> >
> > at java.lang.reflect.Method.invoke(Method.java:597)
> >
> > at com.martiansoftware.nailgun.NGSession.run(NGSession.java:319)
> > 2 i test my branch which updated hive version to org.apache.hive 0.13.1
> > it run successfully when use a bag of 3rd jars as dependency but throw
> > error using assembly jar, it seems assembly jar lead to conflict
> > ERROR DDLTask: java.lang.NoSuchFieldError: doubleTypeInfo
> > at org.apache.hadoop.hive.ql.io.parquet.serde.
> > ArrayWritableObjectInspector.getObjectInspector(
> > ArrayWritableObjectInspector.java:66)
> > at org.apache.hadoop.hive.ql.io.parquet.serde.
> > ArrayWritableObjectInspector.<init>(ArrayWritableObjectInspector.java:59)
> > at org.apache.hadoop.hive.ql.io.parquet.serde.
> > ParquetHiveSerDe.initialize(ParquetHiveSerDe.java:113)
> > at org.apache.hadoop.hive.metastore.MetaStoreUtils.
> > getDeserializer(MetaStoreUtils.java:339)
> > at org.apache.hadoop.hive.ql.metadata.Table.
> > getDeserializerFromMetaStore(Table.java:283)
> > at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(
> > Table.java:189)
> > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(
> > Hive.java:597)
> > at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(
> > DDLTask.java:4194)
> > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.
> > java:281)
> > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> > at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(
> > TaskRunner.java:85)
> >
> >
> >
> >
> >
> > On 2014/9/2 16:45, Sean Owen wrote:
> >
> >> Hm, are you suggesting that the Spark distribution be a bag of 100
> >> JARs? It doesn't quite seem reasonable. It does not remove version
> >> conflicts, just pushes them to run-time, which isn't good. The
> >> assembly is also necessary because that's where shading happens. In
> >> development, you want to run against exactly what will be used in a
> >> real Spark distro.
> >>
> >> On Tue, Sep 2, 2014 at 9:39 AM, scwf <wa...@huawei.com> wrote:
> >>
> >>> hi, all
> >>> I suggest spark not use assembly jar as default run-time
> >>> dependency(spark-submit/spark-class depend on assembly jar),use a
> >>> library of
> >>> all 3rd dependency jar like hadoop/hive/hbase more reasonable.
> >>>
> >>> 1 assembly jar packaged all 3rd jars into a big one, so we need
> >>> rebuild
> >>> this jar if we want to update the version of some component(such as
> >>> hadoop)
> >>> 2 in our practice with spark, sometimes we meet jar compatibility
> >>> issue,
> >>> it is hard to diagnose compatibility issue with assembly jar
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >>> For additional commands, e-mail: dev-help@spark.apache.org
> >>>
> >>>
> >>
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > For additional commands, e-mail: dev-help@spark.apache.org
> >
> >
>
Re: about spark assembly jar
Posted by Sandy Ryza <sa...@cloudera.com>.
This doesn't help for every dependency, but Spark provides an option to
build the assembly jar without Hadoop and its dependencies. We make use of
this in CDH packaging.
-Sandy
On Tue, Sep 2, 2014 at 2:12 AM, scwf <wa...@huawei.com> wrote:
> Hi sean owen,
> here are some problems when i used assembly jar
> 1 i put spark-assembly-*.jar to the lib directory of my application, it
> throw compile error
>
> Error:scalac: Error: class scala.reflect.BeanInfo not found.
> scala.tools.nsc.MissingRequirementError: class scala.reflect.BeanInfo not
> found.
>
> at scala.tools.nsc.symtab.Definitions$definitions$.
> getModuleOrClass(Definitions.scala:655)
>
> at scala.tools.nsc.symtab.Definitions$definitions$.
> getClass(Definitions.scala:608)
>
> at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator.<
> init>(GenJVM.scala:127)
>
> at scala.tools.nsc.backend.jvm.GenJVM$JvmPhase.run(GenJVM.
> scala:85)
>
> at scala.tools.nsc.Global$Run.compileSources(Global.scala:953)
>
> at scala.tools.nsc.Global$Run.compile(Global.scala:1041)
>
> at xsbt.CachedCompiler0.run(CompilerInterface.scala:126)
>
> at xsbt.CachedCompiler0.liftedTree1$1(CompilerInterface.scala:102)
>
> at xsbt.CachedCompiler0.run(CompilerInterface.scala:102)
>
> at xsbt.CompilerInterface.run(CompilerInterface.scala:27)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:39)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> at sbt.compiler.AnalyzingCompiler.call(
> AnalyzingCompiler.scala:102)
>
> at sbt.compiler.AnalyzingCompiler.compile(
> AnalyzingCompiler.scala:48)
>
> at sbt.compiler.AnalyzingCompiler.compile(
> AnalyzingCompiler.scala:41)
>
> at org.jetbrains.jps.incremental.scala.local.
> IdeaIncrementalCompiler.compile(IdeaIncrementalCompiler.scala:28)
>
> at org.jetbrains.jps.incremental.scala.local.LocalServer.
> compile(LocalServer.scala:25)
>
> at org.jetbrains.jps.incremental.scala.remote.Main$.make(Main.
> scala:58)
>
> at org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(
> Main.scala:21)
>
> at org.jetbrains.jps.incremental.scala.remote.Main.nailMain(
> Main.scala)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:39)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> at com.martiansoftware.nailgun.NGSession.run(NGSession.java:319)
> 2 i test my branch which updated hive version to org.apache.hive 0.13.1
> it run successfully when use a bag of 3rd jars as dependency but throw
> error using assembly jar, it seems assembly jar lead to conflict
> ERROR DDLTask: java.lang.NoSuchFieldError: doubleTypeInfo
> at org.apache.hadoop.hive.ql.io.parquet.serde.
> ArrayWritableObjectInspector.getObjectInspector(
> ArrayWritableObjectInspector.java:66)
> at org.apache.hadoop.hive.ql.io.parquet.serde.
> ArrayWritableObjectInspector.<init>(ArrayWritableObjectInspector.java:59)
> at org.apache.hadoop.hive.ql.io.parquet.serde.
> ParquetHiveSerDe.initialize(ParquetHiveSerDe.java:113)
> at org.apache.hadoop.hive.metastore.MetaStoreUtils.
> getDeserializer(MetaStoreUtils.java:339)
> at org.apache.hadoop.hive.ql.metadata.Table.
> getDeserializerFromMetaStore(Table.java:283)
> at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(
> Table.java:189)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(
> Hive.java:597)
> at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(
> DDLTask.java:4194)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.
> java:281)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(
> TaskRunner.java:85)
>
>
>
>
>
> On 2014/9/2 16:45, Sean Owen wrote:
>
>> Hm, are you suggesting that the Spark distribution be a bag of 100
>> JARs? It doesn't quite seem reasonable. It does not remove version
>> conflicts, just pushes them to run-time, which isn't good. The
>> assembly is also necessary because that's where shading happens. In
>> development, you want to run against exactly what will be used in a
>> real Spark distro.
>>
>> On Tue, Sep 2, 2014 at 9:39 AM, scwf <wa...@huawei.com> wrote:
>>
>>> hi, all
>>> I suggest spark not use assembly jar as default run-time
>>> dependency(spark-submit/spark-class depend on assembly jar),use a
>>> library of
>>> all 3rd dependency jar like hadoop/hive/hbase more reasonable.
>>>
>>> 1 assembly jar packaged all 3rd jars into a big one, so we need
>>> rebuild
>>> this jar if we want to update the version of some component(such as
>>> hadoop)
>>> 2 in our practice with spark, sometimes we meet jar compatibility
>>> issue,
>>> it is hard to diagnose compatibility issue with assembly jar
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: about spark assembly jar
Posted by scwf <wa...@huawei.com>.
Hi sean owen,
here are some problems when i used assembly jar
1 i put spark-assembly-*.jar to the lib directory of my application, it throw compile error
Error:scalac: Error: class scala.reflect.BeanInfo not found.
scala.tools.nsc.MissingRequirementError: class scala.reflect.BeanInfo not found.
at scala.tools.nsc.symtab.Definitions$definitions$.getModuleOrClass(Definitions.scala:655)
at scala.tools.nsc.symtab.Definitions$definitions$.getClass(Definitions.scala:608)
at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator.<init>(GenJVM.scala:127)
at scala.tools.nsc.backend.jvm.GenJVM$JvmPhase.run(GenJVM.scala:85)
at scala.tools.nsc.Global$Run.compileSources(Global.scala:953)
at scala.tools.nsc.Global$Run.compile(Global.scala:1041)
at xsbt.CachedCompiler0.run(CompilerInterface.scala:126)
at xsbt.CachedCompiler0.liftedTree1$1(CompilerInterface.scala:102)
at xsbt.CachedCompiler0.run(CompilerInterface.scala:102)
at xsbt.CompilerInterface.run(CompilerInterface.scala:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:102)
at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:48)
at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:41)
at org.jetbrains.jps.incremental.scala.local.IdeaIncrementalCompiler.compile(IdeaIncrementalCompiler.scala:28)
at org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:25)
at org.jetbrains.jps.incremental.scala.remote.Main$.make(Main.scala:58)
at org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:21)
at org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.martiansoftware.nailgun.NGSession.run(NGSession.java:319)
2 i test my branch which updated hive version to org.apache.hive 0.13.1
it run successfully when use a bag of 3rd jars as dependency but throw error using assembly jar, it seems assembly jar lead to conflict
ERROR DDLTask: java.lang.NoSuchFieldError: doubleTypeInfo
at org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectInspector.getObjectInspector(ArrayWritableObjectInspector.java:66)
at org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectInspector.<init>(ArrayWritableObjectInspector.java:59)
at org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe.initialize(ParquetHiveSerDe.java:113)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:339)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:283)
at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:189)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:597)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4194)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:281)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
On 2014/9/2 16:45, Sean Owen wrote:
> Hm, are you suggesting that the Spark distribution be a bag of 100
> JARs? It doesn't quite seem reasonable. It does not remove version
> conflicts, just pushes them to run-time, which isn't good. The
> assembly is also necessary because that's where shading happens. In
> development, you want to run against exactly what will be used in a
> real Spark distro.
>
> On Tue, Sep 2, 2014 at 9:39 AM, scwf <wa...@huawei.com> wrote:
>> hi, all
>> I suggest spark not use assembly jar as default run-time
>> dependency(spark-submit/spark-class depend on assembly jar),use a library of
>> all 3rd dependency jar like hadoop/hive/hbase more reasonable.
>>
>> 1 assembly jar packaged all 3rd jars into a big one, so we need rebuild
>> this jar if we want to update the version of some component(such as hadoop)
>> 2 in our practice with spark, sometimes we meet jar compatibility issue,
>> it is hard to diagnose compatibility issue with assembly jar
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: about spark assembly jar
Posted by scwf <wa...@huawei.com>.
yes, i am not sure what happens when building assembly jar and in my understanding it just package all the dependency jars to a big one.
On 2014/9/2 16:45, Sean Owen wrote:
> Hm, are you suggesting that the Spark distribution be a bag of 100
> JARs? It doesn't quite seem reasonable. It does not remove version
> conflicts, just pushes them to run-time, which isn't good. The
> assembly is also necessary because that's where shading happens. In
> development, you want to run against exactly what will be used in a
> real Spark distro.
>
> On Tue, Sep 2, 2014 at 9:39 AM, scwf <wa...@huawei.com> wrote:
>> hi, all
>> I suggest spark not use assembly jar as default run-time
>> dependency(spark-submit/spark-class depend on assembly jar),use a library of
>> all 3rd dependency jar like hadoop/hive/hbase more reasonable.
>>
>> 1 assembly jar packaged all 3rd jars into a big one, so we need rebuild
>> this jar if we want to update the version of some component(such as hadoop)
>> 2 in our practice with spark, sometimes we meet jar compatibility issue,
>> it is hard to diagnose compatibility issue with assembly jar
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: about spark assembly jar
Posted by Ye Xianjin <ad...@gmail.com>.
Sorry, The quick reply didn't cc the dev list.
Sean, sometimes I have to use the spark-shell to confirm some behavior change. In that case, I have to reassembly the whole project. is there another way around, not use the the big jar in development? For the original question, I have no comments.
--
Ye Xianjin
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Tuesday, September 2, 2014 at 4:58 PM, Sean Owen wrote:
> No, usually you unit-test your changes during development. That
> doesn't require the assembly. Eventually you may wish to test some
> change against the complete assembly.
>
> But that's a different question; I thought you were suggesting that
> the assembly JAR should never be created.
>
> On Tue, Sep 2, 2014 at 9:53 AM, Ye Xianjin <advancedxy@gmail.com (mailto:advancedxy@gmail.com)> wrote:
> > Hi, Sean:
> > In development, do I really need to reassembly the whole project even if I
> > only change a line or two code in one component?
> > I used to that but found time-consuming.
> >
> > --
> > Ye Xianjin
> > Sent with Sparrow
> >
> > On Tuesday, September 2, 2014 at 4:45 PM, Sean Owen wrote:
> >
> > Hm, are you suggesting that the Spark distribution be a bag of 100
> > JARs? It doesn't quite seem reasonable. It does not remove version
> > conflicts, just pushes them to run-time, which isn't good. The
> > assembly is also necessary because that's where shading happens. In
> > development, you want to run against exactly what will be used in a
> > real Spark distro.
> >
> > On Tue, Sep 2, 2014 at 9:39 AM, scwf <wangfei1@huawei.com (mailto:wangfei1@huawei.com)> wrote:
> >
> > hi, all
> > I suggest spark not use assembly jar as default run-time
> > dependency(spark-submit/spark-class depend on assembly jar),use a library of
> > all 3rd dependency jar like hadoop/hive/hbase more reasonable.
> >
> > 1 assembly jar packaged all 3rd jars into a big one, so we need rebuild
> > this jar if we want to update the version of some component(such as hadoop)
> > 2 in our practice with spark, sometimes we meet jar compatibility issue,
> > it is hard to diagnose compatibility issue with assembly jar
> >
> >
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org (mailto:dev-unsubscribe@spark.apache.org)
> > For additional commands, e-mail: dev-help@spark.apache.org (mailto:dev-help@spark.apache.org)
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org (mailto:dev-unsubscribe@spark.apache.org)
> > For additional commands, e-mail: dev-help@spark.apache.org (mailto:dev-help@spark.apache.org)
> >
>
>
>
Re: about spark assembly jar
Posted by Sean Owen <so...@cloudera.com>.
Hm, are you suggesting that the Spark distribution be a bag of 100
JARs? It doesn't quite seem reasonable. It does not remove version
conflicts, just pushes them to run-time, which isn't good. The
assembly is also necessary because that's where shading happens. In
development, you want to run against exactly what will be used in a
real Spark distro.
On Tue, Sep 2, 2014 at 9:39 AM, scwf <wa...@huawei.com> wrote:
> hi, all
> I suggest spark not use assembly jar as default run-time
> dependency(spark-submit/spark-class depend on assembly jar),use a library of
> all 3rd dependency jar like hadoop/hive/hbase more reasonable.
>
> 1 assembly jar packaged all 3rd jars into a big one, so we need rebuild
> this jar if we want to update the version of some component(such as hadoop)
> 2 in our practice with spark, sometimes we meet jar compatibility issue,
> it is hard to diagnose compatibility issue with assembly jar
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org