You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@carbondata.apache.org by Bill Speirs <ws...@datto.com> on 2017/05/24 18:31:13 UTC
Spark 2.1.1 with CarbonData 1.1.0
I'm trying to follow the directions for using Spark 2.1.1 with
CarbonData 1.1.0 found here:
http://carbondata.apache.org/quick-start-guide.html
I compiled CarbonData using: mvn -DskipTests -Pspark-2.1
-Dspark.version=2.1.1 clean package
I ran Spark with: ./bin/spark-shell --jars
../carbondata-parent-1.1.0/assembly/target/scala-2.11/carbondata_2.11-1.1.0-shade-hadoop2.2.0.jar
I imported the two classes (not sure if the second is a typo with the
underline):
scala> import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.SparkSession
scala> import org.apache.spark.sql.CarbonSession._
import org.apache.spark.sql.CarbonSession._
Then tried to create the carbon variable:
scala> val carbon =
SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("hdfs://localhost:9000/carbondata")
<console>:28: error: value getOrCreateCarbonContext is not a member of
org.apache.spark.sql.SparkSession.Builder
val carbon =
SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("hdfs://localhost:9000/carbondata")
Thoughts on what I'm doing wrong here?
Bill-
Re: Spark 2.1.1 with CarbonData 1.1.0
Posted by Ravindra Pesala <ra...@gmail.com>.
Hi Bill,
Actually CarbonData is not supported with spark 2.1.1, it is only supported
with spark 2.1.0.
And about your error it says that method is not present so better give
absolute path of CarbonData jar to spark-shell and also check the imports
are done properly or not. Please refer to the example CarbonSessionExample
inside examples folder for creating carbon session
Regards,
Ravindra
On Thu, 25 May 2017 at 10:30 AM, Pallavi Singh <pa...@knoldus.in>
wrote:
> Hi Bill,
>
> CarbonContext is only for Spark 1.6, for Spark 2.1 you should use
> CarbonSession. Please refer to CarbonSessionExample
> <https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala> for
> more details.
>
> On Thu, May 25, 2017 at 12:01 AM, Bill Speirs <ws...@datto.com> wrote:
>
>> I'm trying to follow the directions for using Spark 2.1.1 with
>> CarbonData 1.1.0 found here:
>> http://carbondata.apache.org/quick-start-guide.html
>>
>> I compiled CarbonData using: mvn -DskipTests -Pspark-2.1
>> -Dspark.version=2.1.1 clean package
>>
>> I ran Spark with: ./bin/spark-shell --jars
>>
>> ../carbondata-parent-1.1.0/assembly/target/scala-2.11/carbondata_2.11-1.1.0-shade-hadoop2.2.0.jar
>>
>> I imported the two classes (not sure if the second is a typo with the
>> underline):
>>
>> scala> import org.apache.spark.sql.SparkSession
>> import org.apache.spark.sql.SparkSession
>>
>> scala> import org.apache.spark.sql.CarbonSession._
>> import org.apache.spark.sql.CarbonSession._
>>
>> Then tried to create the carbon variable:
>>
>> scala> val carbon =
>>
>> SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("hdfs://localhost:9000/carbondata")
>> <console>:28: error: value getOrCreateCarbonContext is not a member of
>> org.apache.spark.sql.SparkSession.Builder
>> val carbon =
>>
>> SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("hdfs://localhost:9000/carbondata")
>>
>> Thoughts on what I'm doing wrong here?
>>
>> Bill-
>>
>
>
>
> --
> Regards | Pallavi Singh
> Software Consultant
> Knoldus Software LLP
> +91-9911235949
>
Re: Spark 2.1.1 with CarbonData 1.1.0
Posted by Pallavi Singh <pa...@knoldus.in>.
Hi Bill,
CarbonContext is only for Spark 1.6, for Spark 2.1 you should use
CarbonSession. Please refer to CarbonSessionExample
<https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala>
for
more details.
On Thu, May 25, 2017 at 12:01 AM, Bill Speirs <ws...@datto.com> wrote:
> I'm trying to follow the directions for using Spark 2.1.1 with
> CarbonData 1.1.0 found here:
> http://carbondata.apache.org/quick-start-guide.html
>
> I compiled CarbonData using: mvn -DskipTests -Pspark-2.1
> -Dspark.version=2.1.1 clean package
>
> I ran Spark with: ./bin/spark-shell --jars
> ../carbondata-parent-1.1.0/assembly/target/scala-2.11/
> carbondata_2.11-1.1.0-shade-hadoop2.2.0.jar
>
> I imported the two classes (not sure if the second is a typo with the
> underline):
>
> scala> import org.apache.spark.sql.SparkSession
> import org.apache.spark.sql.SparkSession
>
> scala> import org.apache.spark.sql.CarbonSession._
> import org.apache.spark.sql.CarbonSession._
>
> Then tried to create the carbon variable:
>
> scala> val carbon =
> SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("
> hdfs://localhost:9000/carbondata")
> <console>:28: error: value getOrCreateCarbonContext is not a member of
> org.apache.spark.sql.SparkSession.Builder
> val carbon =
> SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("
> hdfs://localhost:9000/carbondata")
>
> Thoughts on what I'm doing wrong here?
>
> Bill-
>
--
Regards | Pallavi Singh
Software Consultant
Knoldus Software LLP
+91-9911235949