You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by 李寅威 <25...@qq.com> on 2016/12/15 11:45:01 UTC
Some questions about compiling carbondata
Hi all,
I've tried the following two ways to compile carbondata, but they finally all failed, if anyone could help me:
1. with the latest verson carbondata at github and spark 2.0.0, use the command
mvn -DskipTests -Pspark-2.0 -Dspark.version=2.0.0 clean package
and it compiled successly.
but when I run the script ./bin/carbon-spark-shell , it turns out:
ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
./bin/carbon-spark-shell: line 78: /bin/spark-submit: No such file or directory
and when I run the script ./bin/carbon-spark-sql, it turns out:
$SPARK_HOME is not set
ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
./bin/carbon-spark-sql: line 79: cmd: command not found
./bin/carbon-spark-sql: line 89: /bin/spark-submit: No such file or directory
./bin/carbon-spark-sql: line 89: exec: /bin/spark-submit: cannot execute: No such file or directory
2.with the carbondata-0.2.0 and spark 1.5.0, use the command
mvn -DskipTests -Pspark-1.5 -X -Dspark.version=1.5.0 clean package
and it compiled failed. the error message is :
[ERROR] /home/hadoop/carbondata-0.2.0/integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetadataUtil.scala:46: error: not found: value SqlParser
[INFO] val tableIdent = SqlParser.parseTableIdentifier(tableWithDb)
[INFO] ^
[WARNING] one warning found
[ERROR] one error found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache CarbonData :: Parent ........................ SUCCESS [ 0.971 s]
[INFO] Apache CarbonData :: Common ........................ SUCCESS [ 1.702 s]
[INFO] Apache CarbonData :: Core .......................... SUCCESS [ 4.428 s]
[INFO] Apache CarbonData :: Processing .................... SUCCESS [ 1.587 s]
[INFO] Apache CarbonData :: Hadoop ........................ SUCCESS [ 1.104 s]
[INFO] Apache CarbonData :: Spark ......................... FAILURE [ 11.456 s]
[INFO] Apache CarbonData :: Assembly ...................... SKIPPED
[INFO] Apache CarbonData :: Examples ...................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21.432 s
[INFO] Finished at: 2016-12-15T19:41:29+08:00
[INFO] Final Memory: 69M/995M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:350)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
... 20 more
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:346)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:149)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:136)
at org_scala_tools_maven_executions.JavaMainCallerByFork.run(JavaMainCallerByFork.java:80)
at org_scala_tools_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:124)
at org_scala_tools_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:80)
at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:342)
... 22 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :carbondata-spark
Re: Some questions about compiling carbondata
Posted by Ravindra Pesala <ra...@gmail.com>.
Carbon does not support Spark 2.0, please use Spark 2.1
Use the following command to compile
mvn clean -Pspark-2.1 package
Regards,
Ravindra.
On 15 June 2017 at 14:44, Erlu Chen <ch...@gmail.com> wrote:
> Hi
>
> For Question one, I have raised discussion about carbon-spark-shell for
> spark2.x in following link.
>
> http://apache-carbondata-dev-mailing-list-archive.1130556.
> n5.nabble.com/DISCUSSION-Whether-Carbondata-should-
> keep-carbon-spark-shell-script-td14077.html
>
> Actually there is PR to fix carbon-spark-shell for spark2.x, but I think
> this script is useless.
>
> You can give your opinion, we can vote for this.
>
> Regards.
> Chenerlu
>
>
>
> --
> View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/Some-questions-
> about-compiling-carbondata-tp4498p15139.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>
--
Thanks & Regards,
Ravi
Re: Some questions about compiling carbondata
Posted by Erlu Chen <ch...@gmail.com>.
Hi
For Question one, I have raised discussion about carbon-spark-shell for
spark2.x in following link.
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-Whether-Carbondata-should-keep-carbon-spark-shell-script-td14077.html
Actually there is PR to fix carbon-spark-shell for spark2.x, but I think
this script is useless.
You can give your opinion, we can vote for this.
Regards.
Chenerlu
--
View this message in context: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Some-questions-about-compiling-carbondata-tp4498p15139.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive at Nabble.com.
Re: Some questions about compiling carbondata
Posted by lzzuo2 <10...@qq.com>.
请问这个问题现在解决了么？
--
View this message in context: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Some-questions-about-compiling-carbondata-tp4498p15124.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive at Nabble.com.
Re: Some questions about compiling carbondata
Posted by Sea <26...@qq.com>.
Carbon don't support spark 2.x
------------------ Original ------------------
From: "李寅威";<25...@qq.com>;
Date: Thu, Dec 15, 2016 07:45 PM
To: "dev"<de...@carbondata.incubator.apache.org>;
Subject: Some questions about compiling carbondata
Hi all,
I've tried the following two ways to compile carbondata, but they finally all failed, if anyone could help me:
1. with the latest verson carbondata at github and spark 2.0.0, use the command
mvn -DskipTests -Pspark-2.0 -Dspark.version=2.0.0 clean package
and it compiled successly.
but when I run the script ./bin/carbon-spark-shell , it turns out:
ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
./bin/carbon-spark-shell: line 78: /bin/spark-submit: No such file or directory
and when I run the script ./bin/carbon-spark-sql, it turns out:
$SPARK_HOME is not set
ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
./bin/carbon-spark-sql: line 79: cmd: command not found
./bin/carbon-spark-sql: line 89: /bin/spark-submit: No such file or directory
./bin/carbon-spark-sql: line 89: exec: /bin/spark-submit: cannot execute: No such file or directory
2.with the carbondata-0.2.0 and spark 1.5.0, use the command
mvn -DskipTests -Pspark-1.5 -X -Dspark.version=1.5.0 clean package
and it compiled failed. the error message is :
[ERROR] /home/hadoop/carbondata-0.2.0/integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetadataUtil.scala:46: error: not found: value SqlParser
[INFO] val tableIdent = SqlParser.parseTableIdentifier(tableWithDb)
[INFO] ^
[WARNING] one warning found
[ERROR] one error found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache CarbonData :: Parent ........................ SUCCESS [ 0.971 s]
[INFO] Apache CarbonData :: Common ........................ SUCCESS [ 1.702 s]
[INFO] Apache CarbonData :: Core .......................... SUCCESS [ 4.428 s]
[INFO] Apache CarbonData :: Processing .................... SUCCESS [ 1.587 s]
[INFO] Apache CarbonData :: Hadoop ........................ SUCCESS [ 1.104 s]
[INFO] Apache CarbonData :: Spark ......................... FAILURE [ 11.456 s]
[INFO] Apache CarbonData :: Assembly ...................... SKIPPED
[INFO] Apache CarbonData :: Examples ...................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21.432 s
[INFO] Finished at: 2016-12-15T19:41:29+08:00
[INFO] Final Memory: 69M/995M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:350)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
... 20 more
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:346)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:149)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:136)
at org_scala_tools_maven_executions.JavaMainCallerByFork.run(JavaMainCallerByFork.java:80)
at org_scala_tools_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:124)
at org_scala_tools_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:80)
at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:342)
... 22 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :carbondata-spark
Re: Some questions about compiling carbondata
Posted by Jacky Li <ja...@qq.com>.
Hi,
You do not need to specify spark.version variable, you can try these:
mvn clean package -DskipTests -Pspark-2.0 (to build carbon with
spark-2.0.2)
mvn clean package -DskipTests (to build carbon with spark-1.5.2, which is
default profile)
Regards,
Jacky
--
View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/Some-questions-about-compiling-carbondata-tp4498p4504.html
Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.