You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by watermen <gi...@git.apache.org> on 2017/02/21 06:13:09 UTC
[GitHub] incubator-carbondata pull request #606: [CARBONDATA-713] Make the store path...
GitHub user watermen opened a pull request:
https://github.com/apache/incubator-carbondata/pull/606
[CARBONDATA-713] Make the store path in right priority
The store location of carbon comes from three places:
* default location path in code(../carbon.store)
* configurate "carbon.storelocation" in carbon.properties
* pass the location as the parameter
The priority is low to high.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/watermen/incubator-carbondata CARBONDATA-713
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/606.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #606
----
commit 7b4c2dbf0262d20e894b6820533e63f8ef2a0878
Author: Yadong Qi <qi...@gmail.com>
Date: 2017-02-21T03:01:06Z
Make the store path in right priority.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #606: [CARBONDATA-713] Make the store path in rig...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/606
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/930/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #606: [CARBONDATA-713] Make the store path in rig...
Posted by Hexiaoqiao <gi...@git.apache.org>.
Github user Hexiaoqiao commented on the issue:
https://github.com/apache/incubator-carbondata/pull/606
@chenliang613 @ravipesala it seems not get the item `carbon.storelocation` from carbon.properties if create CarbonContext with `val cc = new CarbonContext(sc)` and always use `./carbon.store` as the secondary parameter when run on cluster mode currently, the detailed exception as maillist mentioned:
http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/template/NamlServlet.jtp?macro=raw_mail&node=7707
pls correct me if i am wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #606: [CARBONDATA-713] Make the store path in rig...
Posted by watermen <gi...@git.apache.org>.
Github user watermen commented on the issue:
https://github.com/apache/incubator-carbondata/pull/606
@ravipesala Can you tell me where is the code about `In cluster we always takes the path from carbon.properties, not from context.`? I only found in SparkEnv.init, we take the path from context.
```scala
def init(sqlContext: SQLContext): Unit = {
if (!initialized) {
val cc = sqlContext.asInstanceOf[CarbonContext]
val catalog = new CarbonMetastore(cc, cc.storePath, cc.hiveClientInterface, "")
carbonEnv = CarbonEnv(catalog)
CarbonIUDAnalysisRule.init(sqlContext)
initialized = true
CarbonProperties.getInstance.addProperty(CarbonCommonConstants.IS_DRIVER_INSTANCE, "true")
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #606: [CARBONDATA-713] Make the store path...
Posted by watermen <gi...@git.apache.org>.
Github user watermen commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/606#discussion_r102370250
--- Diff: integration/spark-common/src/main/scala/org/apache/spark/util/FileUtils.scala ---
@@ -91,4 +94,24 @@ object FileUtils {
}
}
+ /*
+ * The store location of carbon comes from three places:
+ * 1. default location path in code(../carbon.store)
+ * 2. configurate "carbon.storelocation" in carbon.properties
+ * 3. pass the location as the parameter
+ * The priority is low to high.
+ */
+ def getHighestPriorityStorePath(storePathParameter: String): String = {
+ val storePathDefaultValue =
+ new File(CarbonCommonConstants.STORE_LOCATION_DEFAULT_VAL).getCanonicalPath
+ val storePathProperties = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.STORE_LOCATION)
+ if (!storePathParameter.equals(storePathDefaultValue)) {
--- End diff --
@jackylk My first idea is like yours, but I worry about it will change the API. If it can be changed, I will do this solution.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #606: [CARBONDATA-713] Make the store path in rig...
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on the issue:
https://github.com/apache/incubator-carbondata/pull/606
@Hexiaoqiao Yes, you are right, currently do like this what you mentioned above.
I just proposed one solution about how to keep store location consistent between carboncontext and carbon.properties.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #606: [CARBONDATA-713] Make the store path in rig...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/incubator-carbondata/pull/606
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #606: [CARBONDATA-713] Make the store path in rig...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/606
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/929/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #606: [CARBONDATA-713] Make the store path in rig...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/incubator-carbondata/pull/606
@watermen @jackylk we should always give high priority to path provided in carbon.properties. All these issues are because of the paths provided through context and carbon.properties are different.
In cluster we always takes the path from carbon.properties, not from context.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #606: [CARBONDATA-713] Make the store path in rig...
Posted by watermen <gi...@git.apache.org>.
Github user watermen commented on the issue:
https://github.com/apache/incubator-carbondata/pull/606
cc @chenliang613 @jackylk
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #606: [CARBONDATA-713] Make the store path...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/606#discussion_r102372504
--- Diff: integration/spark-common/src/main/scala/org/apache/spark/util/FileUtils.scala ---
@@ -91,4 +94,24 @@ object FileUtils {
}
}
+ /*
+ * The store location of carbon comes from three places:
+ * 1. default location path in code(../carbon.store)
+ * 2. configurate "carbon.storelocation" in carbon.properties
+ * 3. pass the location as the parameter
+ * The priority is low to high.
+ */
+ def getHighestPriorityStorePath(storePathParameter: String): String = {
+ val storePathDefaultValue =
+ new File(CarbonCommonConstants.STORE_LOCATION_DEFAULT_VAL).getCanonicalPath
+ val storePathProperties = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.STORE_LOCATION)
+ if (!storePathParameter.equals(storePathDefaultValue)) {
--- End diff --
I think user can still use `val cc = new CarbonContext(sc)`, so API will not change
@ravipesala what is your opinion
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #606: [CARBONDATA-713] Make the store path in rig...
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on the issue:
https://github.com/apache/incubator-carbondata/pull/606
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #606: [CARBONDATA-713] Make the store path in rig...
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on the issue:
https://github.com/apache/incubator-carbondata/pull/606
@ravipesala as we discussed on yesterday, when user doesn't specify store location(HDFS) in CarbonContext, system if can directly get store location from carbon.properties for keeping consistent,or not ?
----------------------------------------
All these issues are because of the paths provided through context and carbon.properties are different.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #606: [CARBONDATA-713] Make the store path...
Posted by watermen <gi...@git.apache.org>.
Github user watermen commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/606#discussion_r102375559
--- Diff: integration/spark-common/src/main/scala/org/apache/spark/util/FileUtils.scala ---
@@ -91,4 +94,24 @@ object FileUtils {
}
}
+ /*
+ * The store location of carbon comes from three places:
+ * 1. default location path in code(../carbon.store)
+ * 2. configurate "carbon.storelocation" in carbon.properties
+ * 3. pass the location as the parameter
+ * The priority is low to high.
+ */
+ def getHighestPriorityStorePath(storePathParameter: String): String = {
+ val storePathDefaultValue =
+ new File(CarbonCommonConstants.STORE_LOCATION_DEFAULT_VAL).getCanonicalPath
+ val storePathProperties = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.STORE_LOCATION)
+ if (!storePathParameter.equals(storePathDefaultValue)) {
--- End diff --
@jackylk I will keep all of constructors API(`CarbonContext(sc)`, `CarbonContext(sc, storePath)`, `CarbonContext(sc, storePath, metaStorePath)`).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #606: [CARBONDATA-713] Make the store path...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/606#discussion_r102356794
--- Diff: integration/spark-common/src/main/scala/org/apache/spark/util/FileUtils.scala ---
@@ -91,4 +94,24 @@ object FileUtils {
}
}
+ /*
+ * The store location of carbon comes from three places:
+ * 1. default location path in code(../carbon.store)
+ * 2. configurate "carbon.storelocation" in carbon.properties
+ * 3. pass the location as the parameter
+ * The priority is low to high.
+ */
+ def getHighestPriorityStorePath(storePathParameter: String): String = {
+ val storePathDefaultValue =
+ new File(CarbonCommonConstants.STORE_LOCATION_DEFAULT_VAL).getCanonicalPath
+ val storePathProperties = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.STORE_LOCATION)
+ if (!storePathParameter.equals(storePathDefaultValue)) {
--- End diff --
This is strange to me, can you change the constructor of CarbonContext and CarbonSession, to make storePath as an optional parameter whose default value is None. Then you can check weather it is None here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #606: [CARBONDATA-713] Make the store path in rig...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/606
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---