You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crail.apache.org by David Crespi <da...@storedgesystems.com> on 2019/06/19 22:33:09 UTC
Question on wildcard used with spark-default.conf
A question on what is documented on the crail website for configuring to use
the HDFS adapter. The website shows the use of a wildcard (in red):
spark.driver.extraClassPath $CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
spark.executor.extraClassPath $CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
But this didn’t work for me. I had to put all the specific jars onto this path in order for this to work,
such as the lines below. Is there a trick to using the wildcard?
spark.driver.extraClassPath /crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
spark.executor.extraClassPath /crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
This is the error I get with using the wildcard:
Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.crail.hdfs.CrailHadoopFileSystem not found
And if I add that one to the path, then I get the next one it doesn’t find.
Regards,
David
RE: Question on wildcard used with spark-default.conf
Posted by David Crespi <da...@storedgesystems.com>.
Jonas,
Removing the jar from the crail folder worked, it now works with the wildcard!
Progress 😊
Do you still want a ticket filed on this?
Regards,
David
C: 714-476-2692
________________________________
From: Jonas Pfefferle <pe...@japf.ch>
Sent: Tuesday, June 25, 2019 4:16:42 AM
To: David Crespi; dev@crail.apache.org
Subject: Re: Question on wildcard used with spark-default.conf
paranamer is not a direct Crail dependency but is a recursive dependency of
hadoop-common respectively avro.
You can see all the dependencies and dependencies of dependencies by running
"mvn dependency:tree".
I checked if using a newer Hadoop version helps but even for the latest
3.2.0 it still depends on paranamer 2.3. Can you try just deleting the
paranamer jar I doubt that its content is ever loaded since we do not use
the avro file format. I'm not sure what the right solution for this problem
is. Can you please create a JIRA issue so that we can take a look at it.
Regards,
Jonas
On Mon, 24 Jun 2019 21:47:21 +0000
David Crespi <da...@storedgesystems.com> wrote:
> Not sure if you’ve seen this
>(https://jira.apache.org/jira/browse/SPARK-22128)
>
> but it appears that paranamer 2.3 has an issue with Scala 2.12.
> Which is what I’m
>
> running. How do I update crail to use paranamer 2.8… as I mentioned
>before, I’m
>
> not rally that familiar with java.
>
>
> Regards,
>
>
> David
>
>
> C: 714-476-2692
>
>
> ________________________________
>From: Jonas Pfefferle <pe...@japf.ch>
> Sent: Friday, June 21, 2019 12:47:34 AM
> To: David Crespi
> Subject: Re: Question on wildcard used with spark-default.conf
>
> That probably has to do with the fact that you are using Spark 2.4.2
>and
> some of the dependencies used in Spark have different versions in
>Crail.
> Unfortunately, Java does not handle different versions of
>dependencies and
> picks whatever is first in the classpath. What Hadoop version are
>you using
> with Spark? Try using the same with the Crail build. This might fix
>the
> issue. Don't forget to do a "mvn clean" otherwise you might end up
>with
> multiple versions of a jar.
>
>
> Cheers,
> Jonas
>
> On Thu, 20 Jun 2019 14:08:14 +0000
> David Crespi <da...@storedgesystems.com> wrote:
>> When I remove the environment variable and my spark-defaults.conf
>>file contains:
>>
>> spark.driver.extraClassPath /crail/jars/*
>>
>> spark.executor.extraClassPath /crail/jars/*
>>
>>
>> I get the error below. I can’t even used $CRAIL_HOME, as it doesn’t
>>get resolved.
>>
>>
>> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException:
>>17418
>>
>> at
>>com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.accept(BytecodeReadingParanamer.java:532)
>>
>> at
>>com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.access$200(BytecodeReadingParanamer.java:315)
>>
>> at
>>com.thoughtworks.paranamer.BytecodeReadingParanamer.lookupParameterNames(BytecodeReadingParanamer.java:102)
>>
>> at
>>com.thoughtworks.paranamer.CachingParanamer.lookupParameterNames(CachingParanamer.java:76)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.getCtorParams(BeanIntrospector.scala:44)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1(BeanIntrospector.scala:58)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1$adapted(BeanIntrospector.scala:58)
>>
>> at
>>scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:244)
>>
>> at scala.collection.Iterator.foreach(Iterator.scala:941)
>>
>> at scala.collection.Iterator.foreach$(Iterator.scala:941)
>>
>> at
>>scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
>>
>> at
>>scala.collection.IterableLike.foreach(IterableLike.scala:74)
>>
>> at
>>scala.collection.IterableLike.foreach$(IterableLike.scala:73)
>>
>> at
>>scala.collection.AbstractIterable.foreach(Iterable.scala:56)
>>
>> at
>>scala.collection.TraversableLike.flatMap(TraversableLike.scala:244)
>>
>> at
>>scala.collection.TraversableLike.flatMap$(TraversableLike.scala:241)
>>
>> at
>>scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.findConstructorParam$1(BeanIntrospector.scala:58)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$19(BeanIntrospector.scala:176)
>>
>> at
>>scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
>>
>> at
>>scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
>>
>> at
>>scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
>>
>> at
>>scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
>>
>> at
>>scala.collection.TraversableLike.map(TraversableLike.scala:237)
>>
>> at
>>scala.collection.TraversableLike.map$(TraversableLike.scala:230)
>>
>> at
>>scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14(BeanIntrospector.scala:170)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14$adapted(BeanIntrospector.scala:169)
>>
>> at
>>scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:244)
>>
>> at scala.collection.immutable.List.foreach(List.scala:392)
>>
>> at
>>scala.collection.TraversableLike.flatMap(TraversableLike.scala:244)
>>
>> at
>>scala.collection.TraversableLike.flatMap$(TraversableLike.scala:241)
>>
>> at scala.collection.immutable.List.flatMap(List.scala:355)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.apply(BeanIntrospector.scala:169)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$._descriptorFor(ScalaAnnotationIntrospectorModule.scala:22)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.fieldName(ScalaAnnotationIntrospectorModule.scala:30)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.findImplicitPropertyName(ScalaAnnotationIntrospectorModule.scala:78)
>>
>> at
>>com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findImplicitPropertyName(AnnotationIntrospectorPair.java:467)
>>
>> at
>>com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addFields(POJOPropertiesCollector.java:351)
>>
>> at
>>com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:283)
>>
>> at
>>com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueMethod(POJOPropertiesCollector.java:169)
>>
>> at
>>com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueMethod(BasicBeanDescription.java:223)
>>
>> at
>>com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:348)
>>
>> at
>>com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:210)
>>
>> at
>>com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:153)
>>
>> at
>>com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1203)
>>
>> at
>>com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1157)
>>
>> at
>>com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:481)
>>
>> at
>>com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:679)
>>
>> at
>>com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:107)
>>
>> at
>>com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
>>
>> at
>>com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
>>
>> at
>>org.apache.spark.rdd.RDDOperationScope.toJson(RDDOperationScope.scala:52)
>>
>> at
>>org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:145)
>>
>> at
>>org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>>
>> at
>>org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
>>
>> at
>>org.apache.spark.SparkContext.parallelize(SparkContext.scala:716)
>>
>> at
>>com.github.ehiggs.spark.terasort.TeraGen$.main(TeraGen.scala:66)
>>
>> at
>>com.github.ehiggs.spark.terasort.TeraGen.main(TeraGen.scala)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>Method)
>>
>> at
>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:498)
>>
>> at
>>org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>>
>> at
>>org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
>>
>> at
>>org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
>>
>> at
>>org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>>
>> at
>>org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
>>
>> at
>>org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
>>
>> at
>>org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
>>
>> at
>>org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>>
>> Regards,
>>
>>
>> David
>>
>>
>> C: 714-476-2692
>>
>>
>> ________________________________
>>From: Jonas Pfefferle <pe...@japf.ch>
>> Sent: Thursday, June 20, 2019 6:18:32 AM
>> To: David Crespi; dev@crail.apache.org
>> Subject: Re: Question on wildcard used with spark-default.conf
>>
>>For Crail I put all jars in the directory including all dependencies.
>>If you
>> build it from source these will be in assembly/...
>>
>>
>> Regards,
>> Jonas
>>
>> On Thu, 20 Jun 2019 13:16:12 +0000
>> David Crespi <da...@storedgesystems.com> wrote:
>>> I started off doing that way, and I think I was hitting another java
>>>error with something like
>>>
>>> a “number out of bounds” error, as in too many files perhaps. Do
>>>you only put specific files
>>>
>>> into the jar directory? I’ll go back and play around with it again
>>>today and report back.
>>>
>>>
>>> Regards,
>>>
>>>
>>> David
>>>
>>>
>>> ________________________________
>>>From: Jonas Pfefferle <pe...@japf.ch>
>>> Sent: Thursday, June 20, 2019 12:17:36 AM
>>> To: dev@crail.apache.org; David Crespi
>>> Subject: Re: Question on wildcard used with spark-default.conf
>>>
>>> Hi David,
>>>
>>>
>>> the wildcard works for me without problem. However I do not use the
>>> environment variable. Can you try putting the actual Crail path
>>>instead of
>>> the env variable.
>>>
>>> Jonas
>>>
>>> On Wed, 19 Jun 2019 22:33:09 +0000
>>> David Crespi <da...@storedgesystems.com> wrote:
>>>> A question on what is documented on the crail website for
>>>>configuring to use
>>>> the HDFS adapter. The website shows the use of a wildcard (in red):
>>>>
>>>>
>>>> spark.driver.extraClassPath
>>>> $CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
>>>>
>>>> spark.executor.extraClassPath
>>>> $CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
>>>>
>>>> But this didn’t work for me. I had to put all the specific jars
>>>>onto this path in order for this to work,
>>>> such as the lines below. Is there a trick to using the wildcard?
>>>>
>>>> spark.driver.extraClassPath
>>>> /crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
>>>>
>>>> spark.executor.extraClassPath
>>>>/crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
>>>>
>>>> This is the error I get with using the wildcard:
>>>> Exception in thread "main" java.lang.RuntimeException:
>>>>java.lang.ClassNotFoundException: Class
>>>>org.apache.crail.hdfs.CrailHadoopFileSystem not found
>>>>
>>>> And if I add that one to the path, then I get the next one it
>>>>doesn’t find.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> David
>>>>
>>>>
Re: Question on wildcard used with spark-default.conf
Posted by Jonas Pfefferle <pe...@japf.ch>.
paranamer is not a direct Crail dependency but is a recursive dependency of
hadoop-common respectively avro.
You can see all the dependencies and dependencies of dependencies by running
"mvn dependency:tree".
I checked if using a newer Hadoop version helps but even for the latest
3.2.0 it still depends on paranamer 2.3. Can you try just deleting the
paranamer jar I doubt that its content is ever loaded since we do not use
the avro file format. I'm not sure what the right solution for this problem
is. Can you please create a JIRA issue so that we can take a look at it.
Regards,
Jonas
On Mon, 24 Jun 2019 21:47:21 +0000
David Crespi <da...@storedgesystems.com> wrote:
> Not sure if you’ve seen this
>(https://jira.apache.org/jira/browse/SPARK-22128)
>
> but it appears that paranamer 2.3 has an issue with Scala 2.12.
> Which is what I’m
>
> running. How do I update crail to use paranamer 2.8… as I mentioned
>before, I’m
>
> not rally that familiar with java.
>
>
> Regards,
>
>
> David
>
>
> C: 714-476-2692
>
>
> ________________________________
>From: Jonas Pfefferle <pe...@japf.ch>
> Sent: Friday, June 21, 2019 12:47:34 AM
> To: David Crespi
> Subject: Re: Question on wildcard used with spark-default.conf
>
> That probably has to do with the fact that you are using Spark 2.4.2
>and
> some of the dependencies used in Spark have different versions in
>Crail.
> Unfortunately, Java does not handle different versions of
>dependencies and
> picks whatever is first in the classpath. What Hadoop version are
>you using
> with Spark? Try using the same with the Crail build. This might fix
>the
> issue. Don't forget to do a "mvn clean" otherwise you might end up
>with
> multiple versions of a jar.
>
>
> Cheers,
> Jonas
>
> On Thu, 20 Jun 2019 14:08:14 +0000
> David Crespi <da...@storedgesystems.com> wrote:
>> When I remove the environment variable and my spark-defaults.conf
>>file contains:
>>
>> spark.driver.extraClassPath /crail/jars/*
>>
>> spark.executor.extraClassPath /crail/jars/*
>>
>>
>> I get the error below. I can’t even used $CRAIL_HOME, as it doesn’t
>>get resolved.
>>
>>
>> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException:
>>17418
>>
>> at
>>com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.accept(BytecodeReadingParanamer.java:532)
>>
>> at
>>com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.access$200(BytecodeReadingParanamer.java:315)
>>
>> at
>>com.thoughtworks.paranamer.BytecodeReadingParanamer.lookupParameterNames(BytecodeReadingParanamer.java:102)
>>
>> at
>>com.thoughtworks.paranamer.CachingParanamer.lookupParameterNames(CachingParanamer.java:76)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.getCtorParams(BeanIntrospector.scala:44)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1(BeanIntrospector.scala:58)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1$adapted(BeanIntrospector.scala:58)
>>
>> at
>>scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:244)
>>
>> at scala.collection.Iterator.foreach(Iterator.scala:941)
>>
>> at scala.collection.Iterator.foreach$(Iterator.scala:941)
>>
>> at
>>scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
>>
>> at
>>scala.collection.IterableLike.foreach(IterableLike.scala:74)
>>
>> at
>>scala.collection.IterableLike.foreach$(IterableLike.scala:73)
>>
>> at
>>scala.collection.AbstractIterable.foreach(Iterable.scala:56)
>>
>> at
>>scala.collection.TraversableLike.flatMap(TraversableLike.scala:244)
>>
>> at
>>scala.collection.TraversableLike.flatMap$(TraversableLike.scala:241)
>>
>> at
>>scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.findConstructorParam$1(BeanIntrospector.scala:58)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$19(BeanIntrospector.scala:176)
>>
>> at
>>scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
>>
>> at
>>scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
>>
>> at
>>scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
>>
>> at
>>scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
>>
>> at
>>scala.collection.TraversableLike.map(TraversableLike.scala:237)
>>
>> at
>>scala.collection.TraversableLike.map$(TraversableLike.scala:230)
>>
>> at
>>scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14(BeanIntrospector.scala:170)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14$adapted(BeanIntrospector.scala:169)
>>
>> at
>>scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:244)
>>
>> at scala.collection.immutable.List.foreach(List.scala:392)
>>
>> at
>>scala.collection.TraversableLike.flatMap(TraversableLike.scala:244)
>>
>> at
>>scala.collection.TraversableLike.flatMap$(TraversableLike.scala:241)
>>
>> at scala.collection.immutable.List.flatMap(List.scala:355)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.apply(BeanIntrospector.scala:169)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$._descriptorFor(ScalaAnnotationIntrospectorModule.scala:22)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.fieldName(ScalaAnnotationIntrospectorModule.scala:30)
>>
>> at
>>com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.findImplicitPropertyName(ScalaAnnotationIntrospectorModule.scala:78)
>>
>> at
>>com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findImplicitPropertyName(AnnotationIntrospectorPair.java:467)
>>
>> at
>>com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addFields(POJOPropertiesCollector.java:351)
>>
>> at
>>com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:283)
>>
>> at
>>com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueMethod(POJOPropertiesCollector.java:169)
>>
>> at
>>com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueMethod(BasicBeanDescription.java:223)
>>
>> at
>>com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:348)
>>
>> at
>>com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:210)
>>
>> at
>>com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:153)
>>
>> at
>>com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1203)
>>
>> at
>>com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1157)
>>
>> at
>>com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:481)
>>
>> at
>>com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:679)
>>
>> at
>>com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:107)
>>
>> at
>>com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
>>
>> at
>>com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
>>
>> at
>>org.apache.spark.rdd.RDDOperationScope.toJson(RDDOperationScope.scala:52)
>>
>> at
>>org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:145)
>>
>> at
>>org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>>
>> at
>>org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
>>
>> at
>>org.apache.spark.SparkContext.parallelize(SparkContext.scala:716)
>>
>> at
>>com.github.ehiggs.spark.terasort.TeraGen$.main(TeraGen.scala:66)
>>
>> at
>>com.github.ehiggs.spark.terasort.TeraGen.main(TeraGen.scala)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>Method)
>>
>> at
>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:498)
>>
>> at
>>org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>>
>> at
>>org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
>>
>> at
>>org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
>>
>> at
>>org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>>
>> at
>>org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
>>
>> at
>>org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
>>
>> at
>>org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
>>
>> at
>>org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>>
>> Regards,
>>
>>
>> David
>>
>>
>> C: 714-476-2692
>>
>>
>> ________________________________
>>From: Jonas Pfefferle <pe...@japf.ch>
>> Sent: Thursday, June 20, 2019 6:18:32 AM
>> To: David Crespi; dev@crail.apache.org
>> Subject: Re: Question on wildcard used with spark-default.conf
>>
>>For Crail I put all jars in the directory including all dependencies.
>>If you
>> build it from source these will be in assembly/...
>>
>>
>> Regards,
>> Jonas
>>
>> On Thu, 20 Jun 2019 13:16:12 +0000
>> David Crespi <da...@storedgesystems.com> wrote:
>>> I started off doing that way, and I think I was hitting another java
>>>error with something like
>>>
>>> a “number out of bounds” error, as in too many files perhaps. Do
>>>you only put specific files
>>>
>>> into the jar directory? I’ll go back and play around with it again
>>>today and report back.
>>>
>>>
>>> Regards,
>>>
>>>
>>> David
>>>
>>>
>>> ________________________________
>>>From: Jonas Pfefferle <pe...@japf.ch>
>>> Sent: Thursday, June 20, 2019 12:17:36 AM
>>> To: dev@crail.apache.org; David Crespi
>>> Subject: Re: Question on wildcard used with spark-default.conf
>>>
>>> Hi David,
>>>
>>>
>>> the wildcard works for me without problem. However I do not use the
>>> environment variable. Can you try putting the actual Crail path
>>>instead of
>>> the env variable.
>>>
>>> Jonas
>>>
>>> On Wed, 19 Jun 2019 22:33:09 +0000
>>> David Crespi <da...@storedgesystems.com> wrote:
>>>> A question on what is documented on the crail website for
>>>>configuring to use
>>>> the HDFS adapter. The website shows the use of a wildcard (in red):
>>>>
>>>>
>>>> spark.driver.extraClassPath
>>>> $CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
>>>>
>>>> spark.executor.extraClassPath
>>>> $CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
>>>>
>>>> But this didn’t work for me. I had to put all the specific jars
>>>>onto this path in order for this to work,
>>>> such as the lines below. Is there a trick to using the wildcard?
>>>>
>>>> spark.driver.extraClassPath
>>>> /crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
>>>>
>>>> spark.executor.extraClassPath
>>>>/crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
>>>>
>>>> This is the error I get with using the wildcard:
>>>> Exception in thread "main" java.lang.RuntimeException:
>>>>java.lang.ClassNotFoundException: Class
>>>>org.apache.crail.hdfs.CrailHadoopFileSystem not found
>>>>
>>>> And if I add that one to the path, then I get the next one it
>>>>doesn’t find.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> David
>>>>
>>>>
Re: Question on wildcard used with spark-default.conf
Posted by Jonas Pfefferle <pe...@japf.ch>.
For Crail I put all jars in the directory including all dependencies. If you
build it from source these will be in assembly/...
Regards,
Jonas
On Thu, 20 Jun 2019 13:16:12 +0000
David Crespi <da...@storedgesystems.com> wrote:
> I started off doing that way, and I think I was hitting another java
>error with something like
>
> a “number out of bounds” error, as in too many files perhaps. Do
>you only put specific files
>
> into the jar directory? I’ll go back and play around with it again
>today and report back.
>
>
> Regards,
>
>
> David
>
>
> ________________________________
>From: Jonas Pfefferle <pe...@japf.ch>
> Sent: Thursday, June 20, 2019 12:17:36 AM
> To: dev@crail.apache.org; David Crespi
> Subject: Re: Question on wildcard used with spark-default.conf
>
> Hi David,
>
>
> the wildcard works for me without problem. However I do not use the
> environment variable. Can you try putting the actual Crail path
>instead of
> the env variable.
>
> Jonas
>
> On Wed, 19 Jun 2019 22:33:09 +0000
> David Crespi <da...@storedgesystems.com> wrote:
>> A question on what is documented on the crail website for
>>configuring to use
>> the HDFS adapter. The website shows the use of a wildcard (in red):
>>
>>
>> spark.driver.extraClassPath
>> $CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
>>
>> spark.executor.extraClassPath
>> $CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
>>
>> But this didn’t work for me. I had to put all the specific jars
>>onto this path in order for this to work,
>> such as the lines below. Is there a trick to using the wildcard?
>>
>> spark.driver.extraClassPath
>> /crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
>>
>> spark.executor.extraClassPath
>>/crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
>>
>> This is the error I get with using the wildcard:
>> Exception in thread "main" java.lang.RuntimeException:
>>java.lang.ClassNotFoundException: Class
>>org.apache.crail.hdfs.CrailHadoopFileSystem not found
>>
>> And if I add that one to the path, then I get the next one it
>>doesn’t find.
>>
>>
>> Regards,
>>
>> David
>>
>>
Re: Question on wildcard used with spark-default.conf
Posted by Jonas Pfefferle <pe...@japf.ch>.
Hi David,
the wildcard works for me without problem. However I do not use the
environment variable. Can you try putting the actual Crail path instead of
the env variable.
Jonas
On Wed, 19 Jun 2019 22:33:09 +0000
David Crespi <da...@storedgesystems.com> wrote:
> A question on what is documented on the crail website for
>configuring to use
> the HDFS adapter. The website shows the use of a wildcard (in red):
>
>
> spark.driver.extraClassPath
> $CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
>
> spark.executor.extraClassPath
> $CRAIL_HOME/jars/*:$CRAIL_HOME/jars/crail-spark-1.0.jar:.
>
> But this didn’t work for me. I had to put all the specific jars
>onto this path in order for this to work,
> such as the lines below. Is there a trick to using the wildcard?
>
> spark.driver.extraClassPath
> /crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
>
> spark.executor.extraClassPath
>/crail/jars/crail-client-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-hdfs-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-namenode-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-darpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-rpc-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-spark-1.0.jar:/crail/jars/crail-storage-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-narpc-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT-tests.jar:/crail/jars/crail-storage-nvmf-1.2-incubating-SNAPSHOT.jar:/crail/jars/crail-storage-rdma-1.2-incubating-SNAPSHOT.jar:/crail/jars/darpc-1.8.jar:/crail/jars/disni-2.0.jar:/crail/jars/htrace-core-3.1.0-incubating.jar:/crail/jars/narpc-1.4.jar:/jNVMf/target/*:/dinsi/target/*
>
> This is the error I get with using the wildcard:
> Exception in thread "main" java.lang.RuntimeException:
>java.lang.ClassNotFoundException: Class
>org.apache.crail.hdfs.CrailHadoopFileSystem not found
>
> And if I add that one to the path, then I get the next one it
>doesn’t find.
>
>
> Regards,
>
> David
>
>