You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by "sunerhan1992@sina.com" <su...@sina.com> on 2017/06/23 08:49:16 UTC

problem with branch-1.1

hello,
        I tried to use branch-1.1 under hdp2.6.0-spark2.1.0 and met so many problems.
        1.build: use "mvn package -DskipTests -Pspark-2.1 -Dspark.version=2.1.0 -Phadoop-2.7.2" and 
        while compiling core module,throw class not found error,like org.apache.thrift.TBase,
        modify pom.xml in core and add libthrift dependency,compile successfully.
        2.running in spark-shell like ;  
            spark-shell --jars carbondata_2.11-1.1.1-SNAPSHOT-shade-hadoop2.7.2.jar
            scala>import org.apache.spark.sql.SparkSession
            scala>import org.apache.spark.sql.CarbonSession._
            scala>val cc = SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(hdfs path)
            and got error:
            java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/CatalystConf
at org.apache.spark.sql.hive.CarbonSessionState.analyzer$lzycompute(CarbonSessionState.scala:127)
at org.apache.spark.sql.hive.CarbonSessionState.analyzer(CarbonSessionState.scala:126)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
... 52 elided
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.CatalystConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 59 more
check spark-catalyst.jar and didn't found class CatalystConf
3.use apache-spark catalyst jars and rerun with 
spark-shell --jars carbondata_2.11-1.1.1-SNAPSHOT-shade-hadoop2.7.2.jar,spark-catalyst_2.11-2.1.0.jar
and got error message like:
java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/CatalystConf
at org.apache.spark.sql.hive.CarbonSessionState.analyzer$lzycompute(CarbonSessionState.scala:127)
at org.apache.spark.sql.hive.CarbonSessionState.analyzer(CarbonSessionState.scala:126)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
... 52 elided
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.CatalystConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 59 more



sunerhan1992@sina.com

Re: problem with branch-1.1

Posted by David CaiQiang <da...@gmail.com>.
The spark core version of  hdp2.6.0-spark2.1.0 is spark 2.1.1.
In spark 2.1.1, CatalystConf was already removed.

We raised PR to support it and will merge it at later.
https://github.com/apache/carbondata/pull/1096
https://github.com/apache/carbondata/pull/1017

And the command will be "mvn package -DskipTests -Pspark-2.1
-Dspark.version=2.1.1 -Phadoop-2.7.2"



-----
Best Regards
David Cai
--
View this message in context: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/problem-with-branch-1-1-tp16004p16261.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive at Nabble.com.

Re: problem with branch-1.1

Posted by Erlu Chen <ch...@gmail.com>.
The key point is the version of spark and carbondata should match.

Regards.
Chenerlu.



--
View this message in context: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/problem-with-branch-1-1-tp16004p18107.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive at Nabble.com.

Re: problem with branch-1.1

Posted by Mic Sun <su...@sina.com>.
delete aliyun mirror from maven by editing settings.xml file and remove
scala-2.11 from idea's global library even you assigned scala-2.10.



-----
FFCS研究院
--
View this message in context: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/problem-with-branch-1-1-tp16004p18069.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive at Nabble.com.

Re: problem with branch-1.1

Posted by Erlu Chen <ch...@gmail.com>.
Hi,

Please try mvn package -DskipTests -Pspark-2.1 -Dspark.version=2.1.0
-Phadoop-2.7.2 with hadoop2.7.2 and spark 2.

I have just tested, it ok to compile.

[INFO] Reactor Summary:
[INFO] 
[INFO] Apache CarbonData :: Parent ........................ SUCCESS [  1.657
s]
[INFO] Apache CarbonData :: Common ........................ SUCCESS [  1.870
s]
[INFO] Apache CarbonData :: Core .......................... SUCCESS [ 25.003
s]
[INFO] Apache CarbonData :: Processing .................... SUCCESS [  1.941
s]
[INFO] Apache CarbonData :: Hadoop ........................ SUCCESS [  2.017
s]
[INFO] Apache CarbonData :: Spark Common .................. SUCCESS [ 20.622
s]
[INFO] Apache CarbonData :: Spark2 ........................ SUCCESS [ 39.956
s]
[INFO] Apache CarbonData :: Spark Common Test ............. SUCCESS [  4.024
s]
[INFO] Apache CarbonData :: Assembly ...................... SUCCESS [  3.400
s]
[INFO] Apache CarbonData :: Spark2 Examples ............... SUCCESS [  9.718
s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 01:50 min
[INFO] Finished at: 2017-06-23T17:55:28+08:00
[INFO] Final Memory: 83M/860M
[INFO]
------------------------------------------------------------------------
bogon:carbondata erlu$ git branch
* branch-1.1

Regrads.
Chenerlu




--
View this message in context: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/problem-with-branch-1-1-tp16004p16016.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive at Nabble.com.