You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by "sunerhan1992@sina.com" <su...@sina.com> on 2017/06/23 08:49:16 UTC
problem with branch-1.1
hello,
I tried to use branch-1.1 under hdp2.6.0-spark2.1.0 and met so many problems.
1.build: use "mvn package -DskipTests -Pspark-2.1 -Dspark.version=2.1.0 -Phadoop-2.7.2" and
while compiling core module,throw class not found error,like org.apache.thrift.TBase,
modify pom.xml in core and add libthrift dependency,compile successfully.
2.running in spark-shell like ;
spark-shell --jars carbondata_2.11-1.1.1-SNAPSHOT-shade-hadoop2.7.2.jar
scala>import org.apache.spark.sql.SparkSession
scala>import org.apache.spark.sql.CarbonSession._
scala>val cc = SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(hdfs path)
and got error:
java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/CatalystConf
at org.apache.spark.sql.hive.CarbonSessionState.analyzer$lzycompute(CarbonSessionState.scala:127)
at org.apache.spark.sql.hive.CarbonSessionState.analyzer(CarbonSessionState.scala:126)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
... 52 elided
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.CatalystConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 59 more
check spark-catalyst.jar and didn't found class CatalystConf
3.use apache-spark catalyst jars and rerun with
spark-shell --jars carbondata_2.11-1.1.1-SNAPSHOT-shade-hadoop2.7.2.jar,spark-catalyst_2.11-2.1.0.jar
and got error message like:
java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/CatalystConf
at org.apache.spark.sql.hive.CarbonSessionState.analyzer$lzycompute(CarbonSessionState.scala:127)
at org.apache.spark.sql.hive.CarbonSessionState.analyzer(CarbonSessionState.scala:126)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
... 52 elided
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.CatalystConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 59 more
sunerhan1992@sina.com
Re: problem with branch-1.1
Posted by David CaiQiang <da...@gmail.com>.
The spark core version of hdp2.6.0-spark2.1.0 is spark 2.1.1.
In spark 2.1.1, CatalystConf was already removed.
We raised PR to support it and will merge it at later.
https://github.com/apache/carbondata/pull/1096
https://github.com/apache/carbondata/pull/1017
And the command will be "mvn package -DskipTests -Pspark-2.1
-Dspark.version=2.1.1 -Phadoop-2.7.2"
-----
Best Regards
David Cai
--
View this message in context: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/problem-with-branch-1-1-tp16004p16261.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive at Nabble.com.
Re: problem with branch-1.1
Posted by Erlu Chen <ch...@gmail.com>.
The key point is the version of spark and carbondata should match.
Regards.
Chenerlu.
--
View this message in context: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/problem-with-branch-1-1-tp16004p18107.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive at Nabble.com.
Re: problem with branch-1.1
Posted by Mic Sun <su...@sina.com>.
delete aliyun mirror from maven by editing settings.xml file and remove
scala-2.11 from idea's global library even you assigned scala-2.10.
-----
FFCS研究院
--
View this message in context: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/problem-with-branch-1-1-tp16004p18069.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive at Nabble.com.
Re: problem with branch-1.1
Posted by Erlu Chen <ch...@gmail.com>.
Hi,
Please try mvn package -DskipTests -Pspark-2.1 -Dspark.version=2.1.0
-Phadoop-2.7.2 with hadoop2.7.2 and spark 2.
I have just tested, it ok to compile.
[INFO] Reactor Summary:
[INFO]
[INFO] Apache CarbonData :: Parent ........................ SUCCESS [ 1.657
s]
[INFO] Apache CarbonData :: Common ........................ SUCCESS [ 1.870
s]
[INFO] Apache CarbonData :: Core .......................... SUCCESS [ 25.003
s]
[INFO] Apache CarbonData :: Processing .................... SUCCESS [ 1.941
s]
[INFO] Apache CarbonData :: Hadoop ........................ SUCCESS [ 2.017
s]
[INFO] Apache CarbonData :: Spark Common .................. SUCCESS [ 20.622
s]
[INFO] Apache CarbonData :: Spark2 ........................ SUCCESS [ 39.956
s]
[INFO] Apache CarbonData :: Spark Common Test ............. SUCCESS [ 4.024
s]
[INFO] Apache CarbonData :: Assembly ...................... SUCCESS [ 3.400
s]
[INFO] Apache CarbonData :: Spark2 Examples ............... SUCCESS [ 9.718
s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 01:50 min
[INFO] Finished at: 2017-06-23T17:55:28+08:00
[INFO] Final Memory: 83M/860M
[INFO]
------------------------------------------------------------------------
bogon:carbondata erlu$ git branch
* branch-1.1
Regrads.
Chenerlu
--
View this message in context: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/problem-with-branch-1-1-tp16004p16016.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive at Nabble.com.