You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by xuchuanyin <gi...@git.apache.org> on 2017/05/27 08:56:48 UTC
[GitHub] carbondata pull request #964: [CARBONDATA-1099] Fixed bug for carbon-spark-s...
GitHub user xuchuanyin opened a pull request:
https://github.com/apache/carbondata/pull/964
[CARBONDATA-1099] Fixed bug for carbon-spark-shell in spark2 environment
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure the PR title is formatted like:
`[CARBONDATA-<Jira issue #>] Description of pull request`
- [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
- [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
number, if there is one.
- [ ] If this contribution is large, please file an Apache
[Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- What manual testing you have done?
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
---
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xuchuanyin/carbondata update_shell
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/964.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #964
----
commit 7bf23f87c9afd24eb3b711a6ff51bc2339aa3e9a
Author: xuchuanyin <xu...@126.com>
Date: 2017-05-27T08:49:12Z
Update carbon-spark-shell
add search for scala-2.11 folder when compil spark2
commit 3752c2256ef23b66a1a99a89a44db71d9d455c35
Author: xuchuanyin <xu...@126.com>
Date: 2017-05-27T08:52:44Z
Update CarbonSparkILoop.scala
commit fc294bb9204c22ea19b7586e20a1d45812c81915
Author: xuchuanyin <xu...@126.com>
Date: 2017-05-27T08:54:02Z
Create Main.scala
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #964: [CARBONDATA-1099] Fixed bug for carbon-spark-shell in...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/964
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #964: [CARBONDATA-1099] Fixed bug for carbon-spark-s...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/964#discussion_r120515436
--- Diff: integration/spark2/src/main/scala/org/apache/spark/repl/CarbonSparkILoop.scala ---
@@ -19,54 +19,66 @@ package org.apache.spark.repl
class CarbonSparkILoop extends SparkILoop {
- override def initializeSpark() {
- intp.beQuietDuring {
- command("""
- if(org.apache.spark.repl.carbon.Main.interp == null) {
- org.apache.spark.repl.carbon.Main.main(Array[String]())
- }
- """)
- command("val i1 = org.apache.spark.repl.carbon.Main.interp")
- command("import i1._")
- command("""
- @transient val sc = {
- val _sc = i1.createSparkContext()
- println("Spark context available as sc.")
- _sc
- }
- """)
- command("import org.apache.spark.SparkContext._")
- command("import org.apache.spark.sql.CarbonContext")
- command("""
- @transient val cc = {
- val _cc = {
- import java.io.File
- val path = System.getenv("CARBON_HOME") + "/bin/carbonshellstore"
- val store = new File(path)
- store.mkdirs()
- val storePath = sc.getConf.getOption("spark.carbon.storepath")
- .getOrElse(store.getCanonicalPath)
- new CarbonContext(sc, storePath, store.getCanonicalPath)
- }
- println("Carbon context available as cc.")
- _cc
- }
- """)
+ private def initOriginSpark(): Unit = {
+ processLine("""
+ @transient val spark = if (org.apache.spark.repl.Main.sparkSession != null) {
+ org.apache.spark.repl.Main.sparkSession
+ } else {
+ org.apache.spark.repl.Main.createSparkSession()
+ }
+ @transient val sc = {
+ val _sc = spark.sparkContext
+ if (_sc.getConf.getBoolean("spark.ui.reverseProxy", false)) {
+ val proxyUrl = _sc.getConf.get("spark.ui.reverseProxyUrl", null)
+ if (proxyUrl != null) {
+ println(s"Spark Context Web UI is available at " +
+ s"${proxyUrl}/proxy/${_sc.applicationId}")
+ } else {
+ println(s"Spark Context Web UI is available at Spark Master Public URL")
+ }
+ } else {
+ _sc.uiWebUrl.foreach {
+ webUrl => println(s"Spark context Web UI available at ${webUrl}")
+ }
+ }
+ println("Spark context available as 'sc' " +
+ s"(master = ${_sc.master}, app id = ${_sc.applicationId}).")
+ println("Spark session available as 'spark'.")
+ _sc
+ }
+ """)
+ processLine("import org.apache.spark.SparkContext._")
+ processLine("import spark.implicits._")
+ processLine("import spark.sql")
+ processLine("import org.apache.spark.sql.functions._")
+ }
- command("import org.apache.spark.sql.SQLContext")
- command("""
- @transient val sqlContext = {
- val _sqlContext = new SQLContext(sc)
- println("SQL context available as sqlContext.")
- _sqlContext
- }
- """)
- command("import sqlContext.implicits._")
- command("import sqlContext.sql")
+ private def initCarbon(): Unit = {
+ processLine("""
+ import org.apache.spark.sql.SparkSession
+ import org.apache.spark.sql.CarbonSession._
+ @transient val carbon = {
+ val _carbon = {
+ import java.io.File
+ val path = System.getenv("CARBON_HOME") + "/bin/carbonshellstore"
+ val store = new File(path)
+ store.mkdirs()
+ val storePath = sc.getConf.getOption("spark.carbon.storepath")
+ .getOrElse(store.getCanonicalPath)
+ SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(storePath)
--- End diff --
Yes, It works just the same as that in integration/spark1.
Please refer to : https://github.com/apache/carbondata/blob/master/integration/spark/src/main/scala/org/apache/spark/repl/CarbonSparkILoop.scala#L47
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #964: [CARBONDATA-1099] Fixed bug for carbon-spark-s...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/964#discussion_r119761693
--- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java ---
@@ -43,7 +43,7 @@
* Attribute for Carbon LOGGER
*/
private static final LogService LOGGER =
- LogServiceFactory.getLogService(ForwardDictionaryCache.class.getName());
+ LogServiceFactory.getLogService(ReverseDictionaryCache.class.getName());
--- End diff --
yeah, there is no need to fix this in the current issue. I just found it when I submitted the code...
so, should I rollback this change and start a new issue, OR just add another comment in this issue about this change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #964: [CARBONDATA-1099] Fixed bug for carbon-spark-shell in...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/964
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #964: [CARBONDATA-1099] Fixed bug for carbon-spark-s...
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/964#discussion_r119642951
--- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java ---
@@ -43,7 +43,7 @@
* Attribute for Carbon LOGGER
*/
private static final LogService LOGGER =
- LogServiceFactory.getLogService(ForwardDictionaryCache.class.getName());
+ LogServiceFactory.getLogService(ReverseDictionaryCache.class.getName());
--- End diff --
why need to change this code for fix carbon-spark-shell issues?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #964: [CARBONDATA-1099] Fixed bug for carbon-spark-shell in...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/964
carbon-spark-shell has been removed in our discussion in maillist，so we can close this pr.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #964: [CARBONDATA-1099] Fixed bug for carbon-spark-shell in...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/964
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/90/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #964: [CARBONDATA-1099] Fixed bug for carbon-spark-s...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/964#discussion_r120272132
--- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java ---
@@ -43,7 +43,7 @@
* Attribute for Carbon LOGGER
*/
private static final LogService LOGGER =
- LogServiceFactory.getLogService(ForwardDictionaryCache.class.getName());
+ LogServiceFactory.getLogService(ReverseDictionaryCache.class.getName());
--- End diff --
OK, I've already rollback this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #964: [CARBONDATA-1099] Fixed bug for carbon-spark-shell in...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/964
retest
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #964: [CARBONDATA-1099] Fixed bug for carbon-spark-s...
Posted by chenerlu <gi...@git.apache.org>.
Github user chenerlu commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/964#discussion_r120403537
--- Diff: integration/spark2/src/main/scala/org/apache/spark/repl/CarbonSparkILoop.scala ---
@@ -19,54 +19,66 @@ package org.apache.spark.repl
class CarbonSparkILoop extends SparkILoop {
- override def initializeSpark() {
- intp.beQuietDuring {
- command("""
- if(org.apache.spark.repl.carbon.Main.interp == null) {
- org.apache.spark.repl.carbon.Main.main(Array[String]())
- }
- """)
- command("val i1 = org.apache.spark.repl.carbon.Main.interp")
- command("import i1._")
- command("""
- @transient val sc = {
- val _sc = i1.createSparkContext()
- println("Spark context available as sc.")
- _sc
- }
- """)
- command("import org.apache.spark.SparkContext._")
- command("import org.apache.spark.sql.CarbonContext")
- command("""
- @transient val cc = {
- val _cc = {
- import java.io.File
- val path = System.getenv("CARBON_HOME") + "/bin/carbonshellstore"
- val store = new File(path)
- store.mkdirs()
- val storePath = sc.getConf.getOption("spark.carbon.storepath")
- .getOrElse(store.getCanonicalPath)
- new CarbonContext(sc, storePath, store.getCanonicalPath)
- }
- println("Carbon context available as cc.")
- _cc
- }
- """)
+ private def initOriginSpark(): Unit = {
+ processLine("""
+ @transient val spark = if (org.apache.spark.repl.Main.sparkSession != null) {
+ org.apache.spark.repl.Main.sparkSession
+ } else {
+ org.apache.spark.repl.Main.createSparkSession()
+ }
+ @transient val sc = {
+ val _sc = spark.sparkContext
+ if (_sc.getConf.getBoolean("spark.ui.reverseProxy", false)) {
+ val proxyUrl = _sc.getConf.get("spark.ui.reverseProxyUrl", null)
+ if (proxyUrl != null) {
+ println(s"Spark Context Web UI is available at " +
+ s"${proxyUrl}/proxy/${_sc.applicationId}")
+ } else {
+ println(s"Spark Context Web UI is available at Spark Master Public URL")
+ }
+ } else {
+ _sc.uiWebUrl.foreach {
+ webUrl => println(s"Spark context Web UI available at ${webUrl}")
+ }
+ }
+ println("Spark context available as 'sc' " +
+ s"(master = ${_sc.master}, app id = ${_sc.applicationId}).")
+ println("Spark session available as 'spark'.")
+ _sc
+ }
+ """)
+ processLine("import org.apache.spark.SparkContext._")
+ processLine("import spark.implicits._")
+ processLine("import spark.sql")
+ processLine("import org.apache.spark.sql.functions._")
+ }
- command("import org.apache.spark.sql.SQLContext")
- command("""
- @transient val sqlContext = {
- val _sqlContext = new SQLContext(sc)
- println("SQL context available as sqlContext.")
- _sqlContext
- }
- """)
- command("import sqlContext.implicits._")
- command("import sqlContext.sql")
+ private def initCarbon(): Unit = {
+ processLine("""
+ import org.apache.spark.sql.SparkSession
+ import org.apache.spark.sql.CarbonSession._
+ @transient val carbon = {
+ val _carbon = {
+ import java.io.File
+ val path = System.getenv("CARBON_HOME") + "/bin/carbonshellstore"
+ val store = new File(path)
+ store.mkdirs()
+ val storePath = sc.getConf.getOption("spark.carbon.storepath")
+ .getOrElse(store.getCanonicalPath)
+ SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(storePath)
--- End diff --
If you do not specify the metastore path, it will generate carbon.metastore which is in same level with carbondata project.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #964: [CARBONDATA-1099] Fixed bug for carbon-spark-s...
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/964#discussion_r120256136
--- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java ---
@@ -43,7 +43,7 @@
* Attribute for Carbon LOGGER
*/
private static final LogService LOGGER =
- LogServiceFactory.getLogService(ForwardDictionaryCache.class.getName());
+ LogServiceFactory.getLogService(ReverseDictionaryCache.class.getName());
--- End diff --
i suggest that don't need to change this part in this PR. it would be better that each pr clearly fix each issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #964: [CARBONDATA-1099] Fixed bug for carbon-spark-shell in...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/964
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #964: [CARBONDATA-1099] Fixed bug for carbon-spark-s...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin closed the pull request at:
https://github.com/apache/carbondata/pull/964
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #964: [CARBONDATA-1099] Fixed bug for carbon-spark-shell in...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/964
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/96/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #964: [CARBONDATA-1099] Fixed bug for carbon-spark-shell in...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/964
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---