You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by ajantha-bhat <gi...@git.apache.org> on 2018/05/08 12:11:14 UTC
[GitHub] carbondata pull request #2283: [CARBONDATA-2457] Add converter to get Carbon...
GitHub user ajantha-bhat opened a pull request:
https://github.com/apache/carbondata/pull/2283
[CARBONDATA-2457] Add converter to get Carbon SDK Schema from Avro schema directly.
[CARBONDATA-2457] Add converter to get Carbon SDK Schema from Avro schema directly.
In the current implementation, SDK users have to manually create carbon schema of fields from avro schema.
This is time-consuming and error-prone. Also, user should not be worried about this logic.
So, abstract the carbon schema creation from avro schema by exposing a method to user.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed? Added new interface, not modified existing
- [ ] Any backward compatibility impacted? NA
- [ ] Document update required? will be handled in separate PR
- [ ] Testing done
yes, updated the test case in TestNonTransactionalCarbonTable.scala
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ajantha-bhat/carbondata multi_level_complex
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2283.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2283
----
commit a65a6fd0f6663061ddbc7da146e56e8367e75c9f
Author: ajantha-bhat <aj...@...>
Date: 2018-05-08T06:42:37Z
[CARBONDATA-2457] Added converter to get Carbon SDK Schema from Avro schema directly.
In the current implementation, SDK users have to manually create carbon
schema of fields from avro schema.
This is time consuming and error prone. Also user should not be worried
about this logic.
So, abstract the carbon schema creation from avro schema by exposing a
method to user.
----
---
[GitHub] carbondata pull request #2283: [CARBONDATA-2457] Add converter to get Carbon...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2283#discussion_r186935553
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/AvroCarbonWriter.java ---
@@ -118,11 +121,131 @@ private Object avroFieldToObject(Schema.Field avroField, Object fieldValue) {
break;
default:
- throw new UnsupportedOperationException();
+ throw new UnsupportedOperationException(
+ "carbon not support " + type.toString() + " avro type yet");
}
return out;
}
+ /**
+ * converts avro schema to carbon schema required by carbonWriter
+ *
+ * @param avroSchemaString json formatted avro schema as string
+ * @return carbon sdk schema
+ */
+ public static org.apache.carbondata.sdk.file.Schema getCarbonSchemaFromAvroSchema(
+ String avroSchemaString) {
+ if (avroSchemaString == null) {
+ throw new UnsupportedOperationException("avro schema string cannot be null");
+ }
+ Schema avroSchema = new Schema.Parser().parse(avroSchemaString);
+ Field[] carbonField = new Field[avroSchema.getFields().size()];
+ int i = 0;
+ for (Schema.Field avroField : avroSchema.getFields()) {
+ carbonField[i] = prepareFields(avroField.name(), avroField.schema());
+ i++;
+ }
+ return new org.apache.carbondata.sdk.file.Schema(carbonField);
+ }
+
+ private static Field prepareFields(String FieldName, Schema childSchema) {
--- End diff --
Just take avro field, no need to take 2 params
---
[GitHub] carbondata issue #2283: [CARBONDATA-2457] Add converter to get Carbon SDK Sc...
Posted by ajantha-bhat <gi...@git.apache.org>.
Github user ajantha-bhat commented on the issue:
https://github.com/apache/carbondata/pull/2283
@ravipesala : PR is ready
---
[GitHub] carbondata pull request #2283: [CARBONDATA-2457] Add converter to get Carbon...
Posted by ajantha-bhat <gi...@git.apache.org>.
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2283#discussion_r186939573
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/AvroCarbonWriter.java ---
@@ -118,11 +121,131 @@ private Object avroFieldToObject(Schema.Field avroField, Object fieldValue) {
break;
default:
- throw new UnsupportedOperationException();
+ throw new UnsupportedOperationException(
+ "carbon not support " + type.toString() + " avro type yet");
}
return out;
}
+ /**
+ * converts avro schema to carbon schema required by carbonWriter
+ *
+ * @param avroSchemaString json formatted avro schema as string
+ * @return carbon sdk schema
+ */
+ public static org.apache.carbondata.sdk.file.Schema getCarbonSchemaFromAvroSchema(
+ String avroSchemaString) {
+ if (avroSchemaString == null) {
+ throw new UnsupportedOperationException("avro schema string cannot be null");
+ }
+ Schema avroSchema = new Schema.Parser().parse(avroSchemaString);
+ Field[] carbonField = new Field[avroSchema.getFields().size()];
+ int i = 0;
+ for (Schema.Field avroField : avroSchema.getFields()) {
+ carbonField[i] = prepareFields(avroField.name(), avroField.schema());
+ i++;
+ }
+ return new org.apache.carbondata.sdk.file.Schema(carbonField);
+ }
+
+ private static Field prepareFields(String FieldName, Schema childSchema) {
--- End diff --
OK. fixed
---
[GitHub] carbondata pull request #2283: [CARBONDATA-2457] Add converter to get Carbon...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2283
---
[GitHub] carbondata issue #2283: [CARBONDATA-2457] Add converter to get Carbon SDK Sc...
Posted by ajantha-bhat <gi...@git.apache.org>.
Github user ajantha-bhat commented on the issue:
https://github.com/apache/carbondata/pull/2283
retest this please
---
[GitHub] carbondata issue #2283: [CARBONDATA-2457] Add converter to get Carbon SDK Sc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2283
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5748/
---
[GitHub] carbondata issue #2283: [CARBONDATA-2457] Add converter to get Carbon SDK Sc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2283
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4574/
---
[GitHub] carbondata issue #2283: [CARBONDATA-2457] Add converter to get Carbon SDK Sc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2283
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5755/
---
[GitHub] carbondata issue #2283: [CARBONDATA-2457] Add converter to get Carbon SDK Sc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2283
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4596/
---
[GitHub] carbondata issue #2283: [CARBONDATA-2457] Add converter to get Carbon SDK Sc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2283
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4603/
---
[GitHub] carbondata issue #2283: [CARBONDATA-2457] Add converter to get Carbon SDK Sc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2283
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4589/
---
[GitHub] carbondata issue #2283: [CARBONDATA-2457] Add converter to get Carbon SDK Sc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2283
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4816/
---
[GitHub] carbondata issue #2283: [CARBONDATA-2457] Add converter to get Carbon SDK Sc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2283
LGTM
---
[GitHub] carbondata issue #2283: [CARBONDATA-2457] Add converter to get Carbon SDK Sc...
Posted by ajantha-bhat <gi...@git.apache.org>.
Github user ajantha-bhat commented on the issue:
https://github.com/apache/carbondata/pull/2283
retest this please
---
[GitHub] carbondata issue #2283: [CARBONDATA-2457] Add converter to get Carbon SDK Sc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2283
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4823/
---
[GitHub] carbondata issue #2283: [CARBONDATA-2457] Add converter to get Carbon SDK Sc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2283
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5762/
---