You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by xubo245 <gi...@git.apache.org> on 2018/09/20 10:55:19 UTC
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
GitHub user xubo245 opened a pull request:
https://github.com/apache/carbondata/pull/2738
[CARBONDATA-2952] Provide c++ interface for SDK CarbonReader
[CARBONDATA-2952] Provide CarbonReader C++ interface for SDK
1.init carbonreader,config data path and tablename
2.config ak sk endpoing for S3
3.configure projection
4.build carbon reader
5.hasNext
6.readNextRow
7.close
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
Add C++ interface
- [ ] Any backward compatibility impacted?
independency
- [ ] Document update required?
Yes
- [ ] Testing done
test in main
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
this is sub-tasks of https://issues.apache.org/jira/browse/CARBONDATA-2951
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xubo245/carbondata CARBONDATA-2952_CarbonReaderCppAPI
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2738.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2738
----
commit 49735f4f4d714eb76b124e20a1ee6c387b7be6aa
Author: xubo245 <xu...@...>
Date: 2018-09-20T10:35:34Z
[CARBONDATA-2952] Provide CarbonReader C++ interface for SDK
1.init carbonreader,config data path and tablename
2.config ak sk endpoing for S3
3.configure projection
4.build carbon reader
5.hasNext
6.readNextRow
7.close
commit 11ebe2424f8f34d9e9168abcabf216bbce5a8dfa
Author: xubo245 <xu...@...>
Date: 2018-09-20T10:52:14Z
remove the debug folder
----
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/573/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220784150
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReader.java ---
@@ -90,6 +91,33 @@ public T readNextRow() throws IOException, InterruptedException {
return currentReader.getCurrentValue();
}
+ /**
+ * Read and return next string row object
+ */
+ public Object[] readNextStringRow() throws IOException, InterruptedException {
--- End diff --
I don't know in C layer , how to read int, double, date from mix object[] that includes string, int, date. only can read data string now
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220560348
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReader.java ---
@@ -90,6 +91,33 @@ public T readNextRow() throws IOException, InterruptedException {
return currentReader.getCurrentValue();
}
+ /**
+ * Read and return next string row object
+ */
+ public Object[] readNextStringRow() throws IOException, InterruptedException {
--- End diff --
Object conversion to String is not suggested. The representation for the values may change for objects like double, date. So, please use adapter in JNI layer for each of the java data types to c data type
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220459029
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReaderBuilder.java ---
@@ -161,6 +161,20 @@ public CarbonReaderBuilder setEndPoint(String key, String value) {
return this;
}
+
+ public CarbonReaderBuilder withHadoopConf(String[] args) {
--- End diff --
master only support withHadoopConf(Configuration conf), Configuration can't work in C side
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r219873751
--- Diff: store/CSDK/main.cpp ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <stdio.h>
+#include <jni.h>
+#include <stdlib.h>
+#include <iostream>
+#include <unistd.h>
+#include "CarbonReader.h"
+
+using namespace std;
+
+JavaVM *jvm;
+
+JNIEnv *initJVM() {
+ JNIEnv *env;
+ JavaVMInitArgs vm_args;
+ int parNum = 3;
+ int res;
+ JavaVMOption options[parNum];
+
+ options[0].optionString = "-Djava.compiler=NONE";
+ options[1].optionString = "-Djava.class.path=/Users/xubo/Desktop/xubo/git/c/test/jar/carbondata-sdk.jar";
--- End diff --
optimized
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8878/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8871/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2738
retest this please
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220802212
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReaderBuilder.java ---
@@ -101,6 +101,20 @@ public CarbonReaderBuilder withHadoopConf(Configuration conf) {
return this;
}
+
+ public CarbonReaderBuilder withHadoopConf(String[] args) {
--- End diff --
Yes, we can add the withHadoopConf(String key, String value) in java API also.
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220450790
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReaderBuilder.java ---
@@ -161,6 +161,20 @@ public CarbonReaderBuilder setEndPoint(String key, String value) {
return this;
}
+
+ public CarbonReaderBuilder withHadoopConf(String[] args) {
--- End diff --
I think it is added in master already
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220773227
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReaderBuilder.java ---
@@ -101,6 +101,20 @@ public CarbonReaderBuilder withHadoopConf(Configuration conf) {
return this;
}
+
+ public CarbonReaderBuilder withHadoopConf(String[] args) {
+ Configuration configuration = new Configuration();
--- End diff --
this is for replacing withHadoopConf(Configuration conf) in java side, the old configuration will be replaced by new configuration
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220457759
--- Diff: store/sdk/pom.xml ---
@@ -39,6 +39,16 @@
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
--- End diff --
support S3
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/608/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8643/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/2738
Thanks.
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r221218207
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReader.java ---
@@ -90,6 +91,33 @@ public T readNextRow() throws IOException, InterruptedException {
return currentReader.getCurrentValue();
}
+ /**
+ * Read and return next string row object
+ */
+ public Object[] readNextStringRow() throws IOException, InterruptedException {
--- End diff --
OK.. Please add this limitation as only single dimension Array is supported in method signature and remove after enhancing with CarbonRow based API in other PR
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8769/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220833721
--- Diff: store/CSDK/CarbonReader.cpp ---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "CarbonReader.h"
+#include <jni.h>
+
+jobject CarbonReader::builder(JNIEnv *env, char *path, char *tableName) {
+
+ jniEnv = env;
+ jclass carbonReaderClass = env->FindClass("org/apache/carbondata/sdk/file/CarbonReader");
+ jmethodID carbonReaderBuilderID = env->GetStaticMethodID(carbonReaderClass, "builder",
+ "(Ljava/lang/String;Ljava/lang/String;)Lorg/apache/carbondata/sdk/file/CarbonReaderBuilder;");
+ jstring jpath = env->NewStringUTF(path);
+ jstring jtableName = env->NewStringUTF(tableName);
+ jvalue args[2];
+ args[0].l = jpath;
+ args[1].l = jtableName;
+ carbonReaderBuilderObject = env->CallStaticObjectMethodA(carbonReaderClass, carbonReaderBuilderID, args);
+ return carbonReaderBuilderObject;
+}
+
+jobject CarbonReader::projection(int argc, char *argv[]) {
+ jclass carbonReaderBuilderClass = jniEnv->GetObjectClass(carbonReaderBuilderObject);
+ jmethodID buildID = jniEnv->GetMethodID(carbonReaderBuilderClass, "projection",
+ "([Ljava/lang/String;)Lorg/apache/carbondata/sdk/file/CarbonReaderBuilder;");
+ jclass objectArrayClass = jniEnv->FindClass("Ljava/lang/String;");
+ jobjectArray array = jniEnv->NewObjectArray(argc, objectArrayClass, NULL);
+ for (int i = 0; i < argc; ++i) {
+ jstring value = jniEnv->NewStringUTF(argv[i]);
+ jniEnv->SetObjectArrayElement(array, i, value);
+ }
+
+ jvalue args[1];
+ args[0].l = array;
+ carbonReaderBuilderObject = jniEnv->CallObjectMethodA(carbonReaderBuilderObject, buildID, args);
+ return carbonReaderBuilderObject;
+}
+
+jobject CarbonReader::withHadoopConf(int argc, char *argv[]) {
--- End diff --
OK, I added api:config(char *key, char *value)
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220833952
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReaderBuilder.java ---
@@ -101,6 +101,20 @@ public CarbonReaderBuilder withHadoopConf(Configuration conf) {
return this;
}
+
+ public CarbonReaderBuilder withHadoopConf(String[] args) {
--- End diff --
I added api:config(char *key, char *value), only in c layer
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220459945
--- Diff: store/CSDK/main.cpp ---
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <stdio.h>
+#include <jni.h>
+#include <stdlib.h>
+#include <iostream>
+#include <unistd.h>
+#include "CarbonReader.h"
+
+using namespace std;
+
+JavaVM *jvm;
+
+/**
+ * init jvm
+ *
+ * @return
+ */
+JNIEnv *initJVM() {
+ JNIEnv *env;
+ JavaVMInitArgs vm_args;
+ int parNum = 3;
+ int res;
+ JavaVMOption options[parNum];
+
+ options[0].optionString = "-Djava.compiler=NONE";
+ options[1].optionString = "-Djava.class.path=../../sdk/target/carbondata-sdk.jar";
+ options[2].optionString = "-verbose:jni";
+ vm_args.version = JNI_VERSION_1_8;
+ vm_args.nOptions = parNum;
+ vm_args.options = options;
+ vm_args.ignoreUnrecognized = JNI_FALSE;
+
+ res = JNI_CreateJavaVM(&jvm, (void **) &env, &vm_args);
+ if (res < 0) {
+ fprintf(stderr, "\nCan't create Java VM\n");
+ exit(1);
+ }
+
+ return env;
+}
+
+/**
+ * test read data from local disk, without projection
+ *
+ * @param env jni env
+ * @return
+ */
+bool readFromLocalWithoutProjection(JNIEnv *env) {
+
+ CarbonReader carbonReaderClass;
+ carbonReaderClass.builder(env, "../resources/carbondata", "test");
+ carbonReaderClass.build();
+
+ printf("\nRead data from local without projection:\n");
+
+ while (carbonReaderClass.hasNext()) {
+ jobjectArray row = carbonReaderClass.readNextRow();
+ jsize length = env->GetArrayLength(row);
+
+ int j = 0;
+ for (j = 0; j < length; j++) {
+ jobject element = env->GetObjectArrayElement(row, j);
+ char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
+ printf("%s\t", str);
+ }
+ printf("\n");
+ }
+
+ carbonReaderClass.close();
+}
+
+/**
+ * test read data from local disk
+ *
+ * @param env jni env
+ * @return
+ */
+bool readFromLocal(JNIEnv *env) {
+
+ CarbonReader carbonReaderClass;
+ carbonReaderClass.builder(env, "../resources/carbondata", "test");
+
+ char *argv[11];
+ argv[0] = "stringField";
+ argv[1] = "shortField";
+ argv[2] = "intField";
+ argv[3] = "longField";
+ argv[4] = "doubleField";
+ argv[5] = "boolField";
+ argv[6] = "dateField";
+ argv[7] = "timeField";
+ argv[8] = "decimalField";
+ argv[9] = "varcharField";
+ argv[10] = "arrayField";
+ carbonReaderClass.projection(11, argv);
+
+ carbonReaderClass.build();
+
+ printf("\nRead data from local:\n");
+
+ while (carbonReaderClass.hasNext()) {
+ jobjectArray row = carbonReaderClass.readNextRow();
+ jsize length = env->GetArrayLength(row);
+
+ int j = 0;
+ for (j = 0; j < length; j++) {
+ jobject element = env->GetObjectArrayElement(row, j);
+ char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
+ printf("%s\t", str);
+ }
+ printf("\n");
+ }
+
+ carbonReaderClass.close();
+}
+
+
+/**
+ * read data from S3
+ * parameter is ak sk endpoint
+ *
+ * @param env jni env
+ * @param argv argument vector
+ * @return
+ */
+bool readFromS3_2(JNIEnv *env, char *argv[]) {
+ CarbonReader carbonReaderClass;
+
+ char *args[3];
+ // "your access key"
+ args[0] = argv[1];
+ // "your secret key"
+ args[1] = argv[2];
+ // "your endPoint"
+ args[2] = argv[3];
+
+ carbonReaderClass.builder(env, "s3a://sdk/WriterOutput", "test");
+ carbonReaderClass.build(args[0], args[1], args[2]);
+
+ printf("\nRead data from S3:\n");
+ while (carbonReaderClass.hasNext()) {
+ jobjectArray row = carbonReaderClass.readNextRow();
+ jsize length = env->GetArrayLength(row);
+
+ int j = 0;
+ for (j = 0; j < length; j++) {
+ jobject element = env->GetObjectArrayElement(row, j);
+ char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
+ printf("%s\t", str);
+ }
+ printf("\n");
+ }
+
+ carbonReaderClass.close();
+}
+
+/**
+ * read data from S3
+ * parameter is ak sk endpoint
+ *
+ * @param env jni env
+ * @param argv argument vector
+ * @return
+ */
+bool readFromS3(JNIEnv *env, char *argv[]) {
+ CarbonReader carbonReaderClass;
--- End diff --
ok, done
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220450236
--- Diff: store/sdk/pom.xml ---
@@ -39,6 +39,16 @@
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <version>4.2</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
--- End diff --
Why is this required?
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/809/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/2738
@xubo245 Yes, another team of our department use C, now they use C to write parquet file.
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/690/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/602/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220566235
--- Diff: store/CSDK/CarbonReader.cpp ---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "CarbonReader.h"
+#include <jni.h>
+
+jobject CarbonReader::builder(JNIEnv *env, char *path, char *tableName) {
+
+ jniEnv = env;
+ jclass carbonReaderClass = env->FindClass("org/apache/carbondata/sdk/file/CarbonReader");
+ jmethodID carbonReaderBuilderID = env->GetStaticMethodID(carbonReaderClass, "builder",
+ "(Ljava/lang/String;Ljava/lang/String;)Lorg/apache/carbondata/sdk/file/CarbonReaderBuilder;");
+ jstring jpath = env->NewStringUTF(path);
+ jstring jtableName = env->NewStringUTF(tableName);
+ jvalue args[2];
+ args[0].l = jpath;
+ args[1].l = jtableName;
+ carbonReaderBuilderObject = env->CallStaticObjectMethodA(carbonReaderClass, carbonReaderBuilderID, args);
+ return carbonReaderBuilderObject;
+}
+
+jobject CarbonReader::projection(int argc, char *argv[]) {
+ jclass carbonReaderBuilderClass = jniEnv->GetObjectClass(carbonReaderBuilderObject);
+ jmethodID buildID = jniEnv->GetMethodID(carbonReaderBuilderClass, "projection",
+ "([Ljava/lang/String;)Lorg/apache/carbondata/sdk/file/CarbonReaderBuilder;");
+ jclass objectArrayClass = jniEnv->FindClass("Ljava/lang/String;");
+ jobjectArray array = jniEnv->NewObjectArray(argc, objectArrayClass, NULL);
+ for (int i = 0; i < argc; ++i) {
+ jstring value = jniEnv->NewStringUTF(argv[i]);
+ jniEnv->SetObjectArrayElement(array, i, value);
+ }
+
+ jvalue args[1];
+ args[0].l = array;
+ carbonReaderBuilderObject = jniEnv->CallObjectMethodA(carbonReaderBuilderObject, buildID, args);
+ return carbonReaderBuilderObject;
+}
+
+jobject CarbonReader::withHadoopConf(int argc, char *argv[]) {
--- End diff --
As suggested in another comment, let's add simple API to accept Key & value.
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2738
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r221153398
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReader.java ---
@@ -90,6 +91,33 @@ public T readNextRow() throws IOException, InterruptedException {
return currentReader.getCurrentValue();
}
+ /**
+ * Read and return next string row object
+ */
+ public Object[] readNextStringRow() throws IOException, InterruptedException {
--- End diff --
I am developing in local and will raise a new PR for it. CSDK only need read Array[String] now.
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220558655
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReaderBuilder.java ---
@@ -101,6 +101,20 @@ public CarbonReaderBuilder withHadoopConf(Configuration conf) {
return this;
}
+
+ public CarbonReaderBuilder withHadoopConf(String[] args) {
--- End diff --
I think this API is not clear. Instead we can support a method like withHadoopConf(String key, String value) then we don't have to parse in SDK java layer. C JNI layer can be invoked for every property user wants to overwrite
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/652/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220912378
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReaderBuilder.java ---
@@ -101,6 +101,20 @@ public CarbonReaderBuilder withHadoopConf(Configuration conf) {
return this;
}
+
+ public CarbonReaderBuilder withHadoopConf(String[] args) {
+ Configuration configuration = new Configuration();
--- End diff --
I add config(String key, String value) for it.
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/682/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220802829
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReaderBuilder.java ---
@@ -101,6 +101,20 @@ public CarbonReaderBuilder withHadoopConf(Configuration conf) {
return this;
}
+
+ public CarbonReaderBuilder withHadoopConf(String[] args) {
+ Configuration configuration = new Configuration();
--- End diff --
My comment is the same, we should not replace entire conf object every time. Change this method as withHadoopConf(String key, String value) and update property to a class level hadoopConfiguraiton variable.
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/649/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8917/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/618/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2738
please mark it WIP if not ready
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220447008
--- Diff: store/CSDK/main.cpp ---
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <stdio.h>
+#include <jni.h>
+#include <stdlib.h>
+#include <iostream>
+#include <unistd.h>
+#include "CarbonReader.h"
+
+using namespace std;
+
+JavaVM *jvm;
+
+/**
+ * init jvm
+ *
+ * @return
+ */
+JNIEnv *initJVM() {
+ JNIEnv *env;
+ JavaVMInitArgs vm_args;
+ int parNum = 3;
+ int res;
+ JavaVMOption options[parNum];
+
+ options[0].optionString = "-Djava.compiler=NONE";
+ options[1].optionString = "-Djava.class.path=../../sdk/target/carbondata-sdk.jar";
+ options[2].optionString = "-verbose:jni";
+ vm_args.version = JNI_VERSION_1_8;
+ vm_args.nOptions = parNum;
+ vm_args.options = options;
+ vm_args.ignoreUnrecognized = JNI_FALSE;
+
+ res = JNI_CreateJavaVM(&jvm, (void **) &env, &vm_args);
+ if (res < 0) {
+ fprintf(stderr, "\nCan't create Java VM\n");
+ exit(1);
+ }
+
+ return env;
+}
+
+/**
+ * test read data from local disk, without projection
+ *
+ * @param env jni env
+ * @return
+ */
+bool readFromLocalWithoutProjection(JNIEnv *env) {
+
+ CarbonReader carbonReaderClass;
+ carbonReaderClass.builder(env, "../resources/carbondata", "test");
+ carbonReaderClass.build();
+
+ printf("\nRead data from local without projection:\n");
+
+ while (carbonReaderClass.hasNext()) {
+ jobjectArray row = carbonReaderClass.readNextRow();
+ jsize length = env->GetArrayLength(row);
+
+ int j = 0;
+ for (j = 0; j < length; j++) {
+ jobject element = env->GetObjectArrayElement(row, j);
+ char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
+ printf("%s\t", str);
+ }
+ printf("\n");
+ }
+
+ carbonReaderClass.close();
+}
+
+/**
+ * test read data from local disk
+ *
+ * @param env jni env
+ * @return
+ */
+bool readFromLocal(JNIEnv *env) {
+
+ CarbonReader carbonReaderClass;
+ carbonReaderClass.builder(env, "../resources/carbondata", "test");
+
+ char *argv[11];
+ argv[0] = "stringField";
+ argv[1] = "shortField";
+ argv[2] = "intField";
+ argv[3] = "longField";
+ argv[4] = "doubleField";
+ argv[5] = "boolField";
+ argv[6] = "dateField";
+ argv[7] = "timeField";
+ argv[8] = "decimalField";
+ argv[9] = "varcharField";
+ argv[10] = "arrayField";
+ carbonReaderClass.projection(11, argv);
+
+ carbonReaderClass.build();
+
+ printf("\nRead data from local:\n");
+
+ while (carbonReaderClass.hasNext()) {
+ jobjectArray row = carbonReaderClass.readNextRow();
+ jsize length = env->GetArrayLength(row);
+
+ int j = 0;
+ for (j = 0; j < length; j++) {
+ jobject element = env->GetObjectArrayElement(row, j);
+ char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
+ printf("%s\t", str);
+ }
+ printf("\n");
+ }
+
+ carbonReaderClass.close();
+}
+
+
+/**
+ * read data from S3
+ * parameter is ak sk endpoint
+ *
+ * @param env jni env
+ * @param argv argument vector
+ * @return
+ */
+bool readFromS3_2(JNIEnv *env, char *argv[]) {
+ CarbonReader carbonReaderClass;
+
+ char *args[3];
+ // "your access key"
+ args[0] = argv[1];
+ // "your secret key"
+ args[1] = argv[2];
+ // "your endPoint"
+ args[2] = argv[3];
+
+ carbonReaderClass.builder(env, "s3a://sdk/WriterOutput", "test");
+ carbonReaderClass.build(args[0], args[1], args[2]);
+
+ printf("\nRead data from S3:\n");
+ while (carbonReaderClass.hasNext()) {
+ jobjectArray row = carbonReaderClass.readNextRow();
+ jsize length = env->GetArrayLength(row);
+
+ int j = 0;
+ for (j = 0; j < length; j++) {
+ jobject element = env->GetObjectArrayElement(row, j);
+ char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
+ printf("%s\t", str);
+ }
+ printf("\n");
+ }
+
+ carbonReaderClass.close();
+}
+
+/**
+ * read data from S3
+ * parameter is ak sk endpoint
+ *
+ * @param env jni env
+ * @param argv argument vector
+ * @return
+ */
+bool readFromS3(JNIEnv *env, char *argv[]) {
+ CarbonReader carbonReaderClass;
+
+ char *args[3];
+ // "your access key"
+ args[0] = argv[1];
+ // "your secret key"
+ args[1] = argv[2];
+ // "your endPoint"
+ args[2] = argv[3];
+
+ carbonReaderClass.builder(env, "s3a://sdk/WriterOutput", "test");
+ carbonReaderClass.build(3, args);
+
+ printf("\nRead data from S3:\n");
+ while (carbonReaderClass.hasNext()) {
+ jobjectArray row = carbonReaderClass.readNextRow();
+ jsize length = env->GetArrayLength(row);
+
+ int j = 0;
+ for (j = 0; j < length; j++) {
+ jobject element = env->GetObjectArrayElement(row, j);
+ char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
+ printf("%s\t", str);
+ }
+ printf("\n");
+ }
+
+ carbonReaderClass.close();
+}
+
+
+/**
+ * This a example for C++ interface to read carbon file
+ * If you want to test read data fromS3, please input the parameter: ak sk endpoint
+ *
+ * @param argc argument counter
+ * @param argv argument vector
+ * @return
+ */
+int main(int argc, char *argv[]) {
+ // init jvm
+ JNIEnv *env;
+ env = initJVM();
+
+ if (argc > 3) {
+ readFromS3(env, argv);
+ } else {
+ readFromLocal(env);
+ readFromLocalWithoutProjection(env);
+ }
+ cout << "destory jvm\n\n";
+ (jvm)->DestroyJavaVM();
+
+ cout << "\nfinish destory jvm";
+ fprintf(stdout, "Java VM destory.\n");
+ return 0;
+}
+
--- End diff --
There are some binary files added in this PR, please remove them
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220450208
--- Diff: store/sdk/pom.xml ---
@@ -39,6 +39,16 @@
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
--- End diff --
Why is this required?
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/469/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220559298
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReaderBuilder.java ---
@@ -101,6 +101,20 @@ public CarbonReaderBuilder withHadoopConf(Configuration conf) {
return this;
}
+
+ public CarbonReaderBuilder withHadoopConf(String[] args) {
+ Configuration configuration = new Configuration();
--- End diff --
We should not create every time a new configuration. Should create once for first time and then update the value for all further calls to this method
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8945/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/654/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/370/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220907838
--- Diff: store/sdk/pom.xml ---
@@ -39,6 +39,16 @@
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <version>4.2</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
--- End diff --
removed now
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220461925
--- Diff: store/sdk/pom.xml ---
@@ -39,6 +39,16 @@
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <version>4.2</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
--- End diff --
support S3
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2738
retest this please
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r221149564
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReader.java ---
@@ -90,6 +91,33 @@ public T readNextRow() throws IOException, InterruptedException {
return currentReader.getCurrentValue();
}
+ /**
+ * Read and return next string row object
+ */
+ public Object[] readNextStringRow() throws IOException, InterruptedException {
--- End diff --
readNextStringRow is not a standard API I think. It can not work with nested complex data types. Like Array[Array[...]], Struct[Array].
I suggest to add a better API like CarbonRow wrapping on object[] and give reader utility methods like getInt, getString in CarbonRow .
If it can't be finished in this PR scope, temporarily you can move this string making logic to JNI layer code so that it can be improved later. **Adding in java SDK layer will add more confusion to users**
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8879/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220456385
--- Diff: store/CSDK/main.cpp ---
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <stdio.h>
+#include <jni.h>
+#include <stdlib.h>
+#include <iostream>
+#include <unistd.h>
+#include "CarbonReader.h"
+
+using namespace std;
+
+JavaVM *jvm;
+
+/**
+ * init jvm
+ *
+ * @return
+ */
+JNIEnv *initJVM() {
+ JNIEnv *env;
+ JavaVMInitArgs vm_args;
+ int parNum = 3;
+ int res;
+ JavaVMOption options[parNum];
+
+ options[0].optionString = "-Djava.compiler=NONE";
+ options[1].optionString = "-Djava.class.path=../../sdk/target/carbondata-sdk.jar";
+ options[2].optionString = "-verbose:jni";
+ vm_args.version = JNI_VERSION_1_8;
+ vm_args.nOptions = parNum;
+ vm_args.options = options;
+ vm_args.ignoreUnrecognized = JNI_FALSE;
+
+ res = JNI_CreateJavaVM(&jvm, (void **) &env, &vm_args);
+ if (res < 0) {
+ fprintf(stderr, "\nCan't create Java VM\n");
+ exit(1);
+ }
+
+ return env;
+}
+
+/**
+ * test read data from local disk, without projection
+ *
+ * @param env jni env
+ * @return
+ */
+bool readFromLocalWithoutProjection(JNIEnv *env) {
+
+ CarbonReader carbonReaderClass;
+ carbonReaderClass.builder(env, "../resources/carbondata", "test");
+ carbonReaderClass.build();
+
+ printf("\nRead data from local without projection:\n");
+
+ while (carbonReaderClass.hasNext()) {
+ jobjectArray row = carbonReaderClass.readNextRow();
+ jsize length = env->GetArrayLength(row);
+
+ int j = 0;
+ for (j = 0; j < length; j++) {
+ jobject element = env->GetObjectArrayElement(row, j);
+ char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
+ printf("%s\t", str);
+ }
+ printf("\n");
+ }
+
+ carbonReaderClass.close();
+}
+
+/**
+ * test read data from local disk
+ *
+ * @param env jni env
+ * @return
+ */
+bool readFromLocal(JNIEnv *env) {
+
+ CarbonReader carbonReaderClass;
+ carbonReaderClass.builder(env, "../resources/carbondata", "test");
+
+ char *argv[11];
+ argv[0] = "stringField";
+ argv[1] = "shortField";
+ argv[2] = "intField";
+ argv[3] = "longField";
+ argv[4] = "doubleField";
+ argv[5] = "boolField";
+ argv[6] = "dateField";
+ argv[7] = "timeField";
+ argv[8] = "decimalField";
+ argv[9] = "varcharField";
+ argv[10] = "arrayField";
+ carbonReaderClass.projection(11, argv);
+
+ carbonReaderClass.build();
+
+ printf("\nRead data from local:\n");
+
+ while (carbonReaderClass.hasNext()) {
+ jobjectArray row = carbonReaderClass.readNextRow();
+ jsize length = env->GetArrayLength(row);
+
+ int j = 0;
+ for (j = 0; j < length; j++) {
+ jobject element = env->GetObjectArrayElement(row, j);
+ char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
+ printf("%s\t", str);
+ }
+ printf("\n");
+ }
+
+ carbonReaderClass.close();
+}
+
+
+/**
+ * read data from S3
+ * parameter is ak sk endpoint
+ *
+ * @param env jni env
+ * @param argv argument vector
+ * @return
+ */
+bool readFromS3_2(JNIEnv *env, char *argv[]) {
+ CarbonReader carbonReaderClass;
+
+ char *args[3];
+ // "your access key"
+ args[0] = argv[1];
+ // "your secret key"
+ args[1] = argv[2];
+ // "your endPoint"
+ args[2] = argv[3];
+
+ carbonReaderClass.builder(env, "s3a://sdk/WriterOutput", "test");
+ carbonReaderClass.build(args[0], args[1], args[2]);
+
+ printf("\nRead data from S3:\n");
+ while (carbonReaderClass.hasNext()) {
+ jobjectArray row = carbonReaderClass.readNextRow();
+ jsize length = env->GetArrayLength(row);
+
+ int j = 0;
+ for (j = 0; j < length; j++) {
+ jobject element = env->GetObjectArrayElement(row, j);
+ char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
+ printf("%s\t", str);
+ }
+ printf("\n");
+ }
+
+ carbonReaderClass.close();
+}
+
+/**
+ * read data from S3
+ * parameter is ak sk endpoint
+ *
+ * @param env jni env
+ * @param argv argument vector
+ * @return
+ */
+bool readFromS3(JNIEnv *env, char *argv[]) {
+ CarbonReader carbonReaderClass;
+
+ char *args[3];
+ // "your access key"
+ args[0] = argv[1];
+ // "your secret key"
+ args[1] = argv[2];
+ // "your endPoint"
+ args[2] = argv[3];
+
+ carbonReaderClass.builder(env, "s3a://sdk/WriterOutput", "test");
+ carbonReaderClass.build(3, args);
+
+ printf("\nRead data from S3:\n");
+ while (carbonReaderClass.hasNext()) {
+ jobjectArray row = carbonReaderClass.readNextRow();
+ jsize length = env->GetArrayLength(row);
+
+ int j = 0;
+ for (j = 0; j < length; j++) {
+ jobject element = env->GetObjectArrayElement(row, j);
+ char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
+ printf("%s\t", str);
+ }
+ printf("\n");
+ }
+
+ carbonReaderClass.close();
+}
+
+
+/**
+ * This a example for C++ interface to read carbon file
+ * If you want to test read data fromS3, please input the parameter: ak sk endpoint
+ *
+ * @param argc argument counter
+ * @param argv argument vector
+ * @return
+ */
+int main(int argc, char *argv[]) {
+ // init jvm
+ JNIEnv *env;
+ env = initJVM();
+
+ if (argc > 3) {
+ readFromS3(env, argv);
+ } else {
+ readFromLocal(env);
+ readFromLocalWithoutProjection(env);
+ }
+ cout << "destory jvm\n\n";
+ (jvm)->DestroyJavaVM();
+
+ cout << "\nfinish destory jvm";
+ fprintf(stdout, "Java VM destory.\n");
+ return 0;
+}
+
--- End diff --
ok, done
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/395/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/810/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/794/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/786/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8616/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r219700673
--- Diff: store/CSDK/main.cpp ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <stdio.h>
+#include <jni.h>
+#include <stdlib.h>
+#include <iostream>
+#include <unistd.h>
+#include "CarbonReader.h"
+
+using namespace std;
+
+JavaVM *jvm;
+
+JNIEnv *initJVM() {
+ JNIEnv *env;
+ JavaVMInitArgs vm_args;
+ int parNum = 3;
+ int res;
+ JavaVMOption options[parNum];
+
+ options[0].optionString = "-Djava.compiler=NONE";
+ options[1].optionString = "-Djava.class.path=/Users/xubo/Desktop/xubo/git/c/test/jar/carbondata-sdk.jar";
--- End diff --
？？？
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r221423350
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReader.java ---
@@ -90,6 +91,33 @@ public T readNextRow() throws IOException, InterruptedException {
return currentReader.getCurrentValue();
}
+ /**
+ * Read and return next string row object
+ */
+ public Object[] readNextStringRow() throws IOException, InterruptedException {
--- End diff --
OK，I added and will raise new PR for enhancing！
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/849/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220448815
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -506,6 +506,10 @@
* File separator
*/
public static final String FILE_SEPARATOR = "/";
+ /**
+ * ARRAY separator
+ */
+ public static final String ARRAY_SEPARATOR = "\001";
--- End diff --
Can you put this in C code as it is not used in Java side
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on the issue:
https://github.com/apache/carbondata/pull/2738
retest this please
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220774262
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReaderBuilder.java ---
@@ -101,6 +101,20 @@ public CarbonReaderBuilder withHadoopConf(Configuration conf) {
return this;
}
+
+ public CarbonReaderBuilder withHadoopConf(String[] args) {
--- End diff --
yeah. but java layer don't support withHadoopConf(String key, String value), I think we should both support in java and c layer. and config(key: String, value: String) is better than withHadoopConf(String key, String value)
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/610/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/2738
is there a plan for supporting C ?
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/371/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220851174
--- Diff: store/CSDK/CarbonReader.h ---
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef TM_SCALA_CARBONREADER_H
+#define TM_SCALA_CARBONREADER_H
+
+
+#include <jni.h>
+#include <string>
--- End diff --
remove
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/468/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8861/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/524/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/452/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/702/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2738
@zzcclp Now we plan support C++, do you need C interface of carbon?
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8853/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/510/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220451116
--- Diff: store/CSDK/main.cpp ---
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <stdio.h>
+#include <jni.h>
+#include <stdlib.h>
+#include <iostream>
+#include <unistd.h>
+#include "CarbonReader.h"
+
+using namespace std;
+
+JavaVM *jvm;
+
+/**
+ * init jvm
+ *
+ * @return
+ */
+JNIEnv *initJVM() {
+ JNIEnv *env;
+ JavaVMInitArgs vm_args;
+ int parNum = 3;
+ int res;
+ JavaVMOption options[parNum];
+
+ options[0].optionString = "-Djava.compiler=NONE";
+ options[1].optionString = "-Djava.class.path=../../sdk/target/carbondata-sdk.jar";
+ options[2].optionString = "-verbose:jni";
+ vm_args.version = JNI_VERSION_1_8;
+ vm_args.nOptions = parNum;
+ vm_args.options = options;
+ vm_args.ignoreUnrecognized = JNI_FALSE;
+
+ res = JNI_CreateJavaVM(&jvm, (void **) &env, &vm_args);
+ if (res < 0) {
+ fprintf(stderr, "\nCan't create Java VM\n");
+ exit(1);
+ }
+
+ return env;
+}
+
+/**
+ * test read data from local disk, without projection
+ *
+ * @param env jni env
+ * @return
+ */
+bool readFromLocalWithoutProjection(JNIEnv *env) {
+
+ CarbonReader carbonReaderClass;
+ carbonReaderClass.builder(env, "../resources/carbondata", "test");
+ carbonReaderClass.build();
+
+ printf("\nRead data from local without projection:\n");
+
+ while (carbonReaderClass.hasNext()) {
+ jobjectArray row = carbonReaderClass.readNextRow();
+ jsize length = env->GetArrayLength(row);
+
+ int j = 0;
+ for (j = 0; j < length; j++) {
+ jobject element = env->GetObjectArrayElement(row, j);
+ char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
+ printf("%s\t", str);
+ }
+ printf("\n");
+ }
+
+ carbonReaderClass.close();
+}
+
+/**
+ * test read data from local disk
+ *
+ * @param env jni env
+ * @return
+ */
+bool readFromLocal(JNIEnv *env) {
+
+ CarbonReader carbonReaderClass;
+ carbonReaderClass.builder(env, "../resources/carbondata", "test");
+
+ char *argv[11];
+ argv[0] = "stringField";
+ argv[1] = "shortField";
+ argv[2] = "intField";
+ argv[3] = "longField";
+ argv[4] = "doubleField";
+ argv[5] = "boolField";
+ argv[6] = "dateField";
+ argv[7] = "timeField";
+ argv[8] = "decimalField";
+ argv[9] = "varcharField";
+ argv[10] = "arrayField";
+ carbonReaderClass.projection(11, argv);
+
+ carbonReaderClass.build();
+
+ printf("\nRead data from local:\n");
+
+ while (carbonReaderClass.hasNext()) {
+ jobjectArray row = carbonReaderClass.readNextRow();
+ jsize length = env->GetArrayLength(row);
+
+ int j = 0;
+ for (j = 0; j < length; j++) {
+ jobject element = env->GetObjectArrayElement(row, j);
+ char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
+ printf("%s\t", str);
+ }
+ printf("\n");
+ }
+
+ carbonReaderClass.close();
+}
+
+
+/**
+ * read data from S3
+ * parameter is ak sk endpoint
+ *
+ * @param env jni env
+ * @param argv argument vector
+ * @return
+ */
+bool readFromS3_2(JNIEnv *env, char *argv[]) {
+ CarbonReader carbonReaderClass;
+
+ char *args[3];
+ // "your access key"
+ args[0] = argv[1];
+ // "your secret key"
+ args[1] = argv[2];
+ // "your endPoint"
+ args[2] = argv[3];
+
+ carbonReaderClass.builder(env, "s3a://sdk/WriterOutput", "test");
+ carbonReaderClass.build(args[0], args[1], args[2]);
+
+ printf("\nRead data from S3:\n");
+ while (carbonReaderClass.hasNext()) {
+ jobjectArray row = carbonReaderClass.readNextRow();
+ jsize length = env->GetArrayLength(row);
+
+ int j = 0;
+ for (j = 0; j < length; j++) {
+ jobject element = env->GetObjectArrayElement(row, j);
+ char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
+ printf("%s\t", str);
+ }
+ printf("\n");
+ }
+
+ carbonReaderClass.close();
+}
+
+/**
+ * read data from S3
+ * parameter is ak sk endpoint
+ *
+ * @param env jni env
+ * @param argv argument vector
+ * @return
+ */
+bool readFromS3(JNIEnv *env, char *argv[]) {
+ CarbonReader carbonReaderClass;
--- End diff --
rename carbonReaderClass to reader
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/546/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/368/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/802/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on the issue:
https://github.com/apache/carbondata/pull/2738
LGTM.
One minor comment -> Reference for CSDK-guide.md is missing from README due to which there is no way to navigate to the guide. Please handle this as part of #2792
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8719/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/548/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220565596
--- Diff: store/sdk/pom.xml ---
@@ -39,6 +39,16 @@
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <version>4.2</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
--- End diff --
carbondata-hadoop already depends on hadoop. Why to add explicitly here?
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2738
ok, I will try to support C to write carbon file later.
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8756/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/632/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/450/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2738
@KanakaKumar @kunal642 @jackylk @ravikiran23 Can you review it?
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/617/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220912011
--- Diff: store/sdk/pom.xml ---
@@ -39,6 +39,16 @@
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
--- End diff --
ok, done.support hadoop 2.7.2 and 2.8.3
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8703/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by KanakaKumar <gi...@git.apache.org>.
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220562265
--- Diff: store/sdk/pom.xml ---
@@ -39,6 +39,16 @@
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
--- End diff --
httpclient version number dependent may be different in different hadoop versions. So, it should be downloaded as part of carbon-parent pom. Can you verify & avoid this ?
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/877/
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2738
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8618/
---
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r220784376
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReader.java ---
@@ -90,6 +91,33 @@ public T readNextRow() throws IOException, InterruptedException {
return currentReader.getCurrentValue();
}
+ /**
+ * Read and return next string row object
+ */
+ public Object[] readNextStringRow() throws IOException, InterruptedException {
--- End diff --
I convert double and date to string first, then return string[]. So the values didn't change.
---