You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by xubo245 <gi...@git.apache.org> on 2018/11/20 07:39:08 UTC
[GitHub] carbondata pull request #2931: [CARBONDATA-2999] support read schema from S3
GitHub user xubo245 opened a pull request:
https://github.com/apache/carbondata/pull/2931
[CARBONDATA-2999] support read schema from S3
[CARBONDATA-2999] support read schema from S3
SDK and CSDK both support it.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
add
- [ ] Any backward compatibility impacted?
No
- [ ] Document update required?
Yes
- [ ] Testing done
add test case
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
JIRA2951
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xubo245/carbondata CARBONDATA-2999_readSchemaFromS3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2931.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2931
----
commit 1d5246dd33b743de68aae6ef872623aca8bcd964
Author: xubo245 <xu...@...>
Date: 2018-11-20T07:36:43Z
[CARBONDATA-2999] support read schema from S3
----
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10070/
---
[GitHub] carbondata pull request #2931: [CARBONDATA-2999] support read schema from S3
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2931#discussion_r242428274
--- Diff: store/CSDK/test/main.cpp ---
@@ -822,16 +831,17 @@ int main(int argc, char *argv[]) {
// init jvm
JNIEnv *env;
env = initJVM();
- char *S3WritePath = "s3a://csdk/WriterOutput/carbondata2";
- char *S3ReadPath = "s3a://csdk/WriterOutput/carbondata";
+ char *S3WritePath = "s3a://xubo/WriterOutput/carbondata2";
--- End diff --
please change the path
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1501/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1825/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2021/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9874/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2931
@KanakaKumar @jackylk @QiangCai @ajantha-bhat @kunal642 Rebased, Please review it.
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1711/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1667/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2931
@KanakaKumar @BJangir @ajantha-bhat @jackylk Please review it.
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9716/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1614/
---
[GitHub] carbondata pull request #2931: [CARBONDATA-2999] support read schema from S3
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2931#discussion_r242034641
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonSchemaReader.java ---
@@ -147,34 +170,55 @@ public static Schema readSchema(String path, boolean validateSchema) throws IOEx
throw new CarbonDataLoadingException("No carbonindex file in this path.");
}
} else {
- String indexFilePath = getCarbonFile(path, INDEX_FILE_EXT)[0].getAbsolutePath();
- return readSchemaFromIndexFile(indexFilePath);
+ String indexFilePath = getCarbonFile(path, INDEX_FILE_EXT, conf)[0].getAbsolutePath();
+ return readSchemaFromIndexFile(indexFilePath, conf);
}
}
+ /**
+ * read schema from path,
+ * path can be folder path, carbonindex file path, and carbondata file path
+ * and user can decide whether check all files schema
+ *
+ * @param path file/folder path
+ * @param validateSchema whether check all files schema
+ * @return schema
+ * @throws IOException
+ */
+ public static Schema readSchema(String path, boolean validateSchema) throws IOException {
+ Configuration conf = new Configuration();
+ return readSchema(path, validateSchema, conf);
+ }
+
/**
* Read carbondata file and return the schema
* This interface will be removed,
* please use readSchema instead of this interface
*
* @param dataFilePath carbondata file store path
+ * @param conf hadoop configuration support, can set s3a AK,SK,
+ * end point and other conf with this
* @return Schema object
* @throws IOException
*/
@Deprecated
- public static Schema readSchemaInDataFile(String dataFilePath) throws IOException {
- return readSchema(dataFilePath, false);
+ public static Schema readSchemaInDataFile(String dataFilePath, Configuration conf)
--- End diff --
No need to change the deprecated old method. Keep as it is.
---
[GitHub] carbondata pull request #2931: [CARBONDATA-2999] support read schema from S3
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2931
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10075/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1668/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1819/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9759/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9715/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1816/
---
[GitHub] carbondata pull request #2931: [CARBONDATA-2999] support read schema from S3
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2931#discussion_r242387341
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonSchemaReader.java ---
@@ -147,34 +170,55 @@ public static Schema readSchema(String path, boolean validateSchema) throws IOEx
throw new CarbonDataLoadingException("No carbonindex file in this path.");
}
} else {
- String indexFilePath = getCarbonFile(path, INDEX_FILE_EXT)[0].getAbsolutePath();
- return readSchemaFromIndexFile(indexFilePath);
+ String indexFilePath = getCarbonFile(path, INDEX_FILE_EXT, conf)[0].getAbsolutePath();
+ return readSchemaFromIndexFile(indexFilePath, conf);
}
}
+ /**
+ * read schema from path,
+ * path can be folder path, carbonindex file path, and carbondata file path
+ * and user can decide whether check all files schema
+ *
+ * @param path file/folder path
+ * @param validateSchema whether check all files schema
+ * @return schema
+ * @throws IOException
+ */
+ public static Schema readSchema(String path, boolean validateSchema) throws IOException {
+ Configuration conf = new Configuration();
+ return readSchema(path, validateSchema, conf);
+ }
+
/**
* Read carbondata file and return the schema
* This interface will be removed,
* please use readSchema instead of this interface
*
* @param dataFilePath carbondata file store path
+ * @param conf hadoop configuration support, can set s3a AK,SK,
+ * end point and other conf with this
* @return Schema object
* @throws IOException
*/
@Deprecated
- public static Schema readSchemaInDataFile(String dataFilePath) throws IOException {
- return readSchema(dataFilePath, false);
+ public static Schema readSchemaInDataFile(String dataFilePath, Configuration conf)
--- End diff --
ok, done
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2028/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1457/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2931
@KanakaKumar @jackylk @QiangCai @ajantha-bhat @kunal642 Rebased, Please review it.
---
[GitHub] carbondata pull request #2931: [CARBONDATA-2999] support read schema from S3
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2931#discussion_r242444376
--- Diff: store/CSDK/test/main.cpp ---
@@ -822,16 +831,17 @@ int main(int argc, char *argv[]) {
// init jvm
JNIEnv *env;
env = initJVM();
- char *S3WritePath = "s3a://csdk/WriterOutput/carbondata2";
- char *S3ReadPath = "s3a://csdk/WriterOutput/carbondata";
+ char *S3WritePath = "s3a://xubo/WriterOutput/carbondata2";
--- End diff --
OKļ¼done
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on the issue:
https://github.com/apache/carbondata/pull/2931
LGTM
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on the issue:
https://github.com/apache/carbondata/pull/2931
LGTM
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2931
rebase and fix the conflict
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1811/
---
[GitHub] carbondata issue #2931: [CARBONDATA-2999] support read schema from S3
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2931
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10077/
---