You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by sounakr <gi...@git.apache.org> on 2018/04/23 04:53:14 UTC
[GitHub] carbondata pull request #2209: [WIP][Non Transactional Table]Avro Record Com...
GitHub user sounakr opened a pull request:
https://github.com/apache/carbondata/pull/2209
[WIP][Non Transactional Table]Avro Record Complex Type Implementation
Avro Complex DataType Support.
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sounakr/incubator-carbondata avro-datatype
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2209.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2209
----
commit 464ade68ca1d45a4fcac15c070b7edae8838bff7
Author: sounakr <so...@...>
Date: 2018-04-23T04:48:10Z
Avro Record Complex Type Implementation
----
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5521/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2209
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184976155
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
dataOutputStream.write(v);
}
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
--- End diff --
Removed
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184975993
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
dataOutputStream.write(v);
}
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
--- End diff --
Removed
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184919245
--- Diff: processing/src/main/java/org/apache/carbondata/processing/store/CarbonFactDataHandlerModel.java ---
@@ -609,5 +616,13 @@ public short getWritingCoresCount() {
public DataMapWriterListener getDataMapWriterlistener() {
return dataMapWriterlistener;
}
+
+ public Boolean[][] getComplexDictionaryFields() {
--- End diff --
change this to boolean[][] getComplexDictionaryFields()
---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4354/
---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2209
Please modify the PR title, it is for SDK, change to [SDK]
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r185217438
--- Diff: integration/spark2/src/main/scala/org/apache/spark/util/SparkTypeConverter.scala ---
@@ -123,13 +137,29 @@ private[spark] object SparkTypeConverter {
private def recursiveMethod(
table: CarbonTable, dimName: String, childDim: CarbonDimension) = {
childDim.getDataType.getName.toLowerCase match {
- case "array" => s"${
- childDim.getColName.substring(dimName.length + 1)
- }:array<${ getArrayChildren(table, childDim.getColName) }>"
- case "struct" => s"${
- childDim.getColName.substring(dimName.length + 1)
- }:struct<${ getStructChildren(table, childDim.getColName) }>"
- case dType => s"${ childDim.getColName.substring(dimName.length + 1) }:${ dType }"
+ case "array" => if (table.isTransactionalTable) {
--- End diff --
Or else we can make the behavior of transactional or non transactional same.
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184975951
--- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/CarbonDataLoadConfiguration.java ---
@@ -348,6 +356,33 @@ public void setCardinalityFinder(DictionaryCardinalityFinder cardinalityFinder)
return complexKeyGenerators;
}
+ public Boolean[][] createComplexDictionaryFieldIdentification() {
--- End diff --
Removed
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5542/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918635
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -211,29 +241,66 @@ public int getSurrogateIndex() {
/*
* set surrogate index
*/
- @Override
- public void setSurrogateIndex(int surrIndex) {
- index = surrIndex;
+ @Override public void setSurrogateIndex(int surrIndex) {
+ if (this.carbonDimension != null && !this.carbonDimension.hasEncoding(Encoding.DICTIONARY)) {
+ index = 0;
+ } else if (this.carbonDimension == null && isDictionary == false) {
+ index = 0;
+ } else {
+ index = surrIndex;
+ }
+ }
+
+ @Override public Boolean getIsColumnDictionary() {
+ return isDictionary;
}
@Override public void writeByteArray(Object input, DataOutputStream dataOutputStream)
throws IOException, DictionaryGenerationException {
String parsedValue =
input == null ? null : DataTypeUtil.parseValue(input.toString(), carbonDimension);
- Integer surrogateKey;
- if (null == parsedValue) {
- surrogateKey = CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
- } else {
- surrogateKey = dictionaryGenerator.getOrGenerateKey(parsedValue);
- if (surrogateKey == CarbonCommonConstants.INVALID_SURROGATE_KEY) {
+ if (this.isDictionary) {
+ Integer surrogateKey;
+ if (null == parsedValue) {
surrogateKey = CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
+ } else {
+ surrogateKey = dictionaryGenerator.getOrGenerateKey(parsedValue);
+ if (surrogateKey == CarbonCommonConstants.INVALID_SURROGATE_KEY) {
+ surrogateKey = CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
+ }
+ }
+ dataOutputStream.writeInt(surrogateKey);
+ } else {
+ // Transform into ByteArray for No Dictionary.
+ // TODO have to refactor and place all the cases present in NonDictionaryFieldConverterImpl
+ if (null == parsedValue && this.carbonDimension.getDataType() != DataTypes.STRING) {
+ updateNullValue(dataOutputStream);
+ } else {
+ byte[] value = DataTypeUtil.getBytesBasedOnDataTypeForNoDictionaryColumn(parsedValue,
--- End diff --
In case of no dictionary primitive type column NoDictionary field converter will convert and give the parsed value so no need to call DataTypeUtil.getBytesBasedOnDataTypeForNoDictionaryColumn again, dictionary write this value in LV format to outputstream
---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5519/
---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4621/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184976110
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
dataOutputStream.write(v);
}
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
+ throws IOException, KeyGenException {
+ if (!this.isDictionary) {
+ int sizeOfData = byteArrayInput.getInt();
+ startOffset += Integer.SIZE / Byte.SIZE;
--- End diff --
Added the constant
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184964228
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -94,6 +99,11 @@
private CarbonDimension carbonDimension;
+ private Boolean isDictionary;
--- End diff --
done
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184976615
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
@@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
for (int i = 0; i < dataLength; i++) {
children.parseAndBitPack(byteArrayInput, dataOutputStream, generator);
}
+ }
+
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
--- End diff --
removed complexDictionaryIndentification
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4398/
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
LGTM
---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4611/
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4357/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184919017
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
dataOutputStream.write(v);
}
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
+ throws IOException, KeyGenException {
+ if (!this.isDictionary) {
+ int sizeOfData = byteArrayInput.getInt();
+ startOffset += Integer.SIZE / Byte.SIZE;
--- End diff --
No need to calculate every time , just use 4 and add comment , Do it in all places
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918197
--- Diff: integration/spark2/src/main/scala/org/apache/spark/util/SparkTypeConverter.scala ---
@@ -123,13 +137,29 @@ private[spark] object SparkTypeConverter {
private def recursiveMethod(
table: CarbonTable, dimName: String, childDim: CarbonDimension) = {
childDim.getDataType.getName.toLowerCase match {
- case "array" => s"${
- childDim.getColName.substring(dimName.length + 1)
- }:array<${ getArrayChildren(table, childDim.getColName) }>"
- case "struct" => s"${
- childDim.getColName.substring(dimName.length + 1)
- }:struct<${ getStructChildren(table, childDim.getColName) }>"
- case dType => s"${ childDim.getColName.substring(dimName.length + 1) }:${ dType }"
+ case "array" => if (table.isTransactionalTable) {
--- End diff --
Better write another method rather than keeping so many if checks
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918097
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -211,29 +241,66 @@ public int getSurrogateIndex() {
/*
* set surrogate index
*/
- @Override
- public void setSurrogateIndex(int surrIndex) {
- index = surrIndex;
+ @Override public void setSurrogateIndex(int surrIndex) {
+ if (this.carbonDimension != null && !this.carbonDimension.hasEncoding(Encoding.DICTIONARY)) {
+ index = 0;
+ } else if (this.carbonDimension == null && isDictionary == false) {
+ index = 0;
+ } else {
+ index = surrIndex;
+ }
+ }
+
+ @Override public Boolean getIsColumnDictionary() {
+ return isDictionary;
}
@Override public void writeByteArray(Object input, DataOutputStream dataOutputStream)
throws IOException, DictionaryGenerationException {
String parsedValue =
input == null ? null : DataTypeUtil.parseValue(input.toString(), carbonDimension);
- Integer surrogateKey;
- if (null == parsedValue) {
- surrogateKey = CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
- } else {
- surrogateKey = dictionaryGenerator.getOrGenerateKey(parsedValue);
- if (surrogateKey == CarbonCommonConstants.INVALID_SURROGATE_KEY) {
+ if (this.isDictionary) {
+ Integer surrogateKey;
+ if (null == parsedValue) {
surrogateKey = CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
+ } else {
+ surrogateKey = dictionaryGenerator.getOrGenerateKey(parsedValue);
+ if (surrogateKey == CarbonCommonConstants.INVALID_SURROGATE_KEY) {
+ surrogateKey = CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
+ }
+ }
+ dataOutputStream.writeInt(surrogateKey);
+ } else {
+ // Transform into ByteArray for No Dictionary.
+ // TODO have to refactor and place all the cases present in NonDictionaryFieldConverterImpl
+ if (null == parsedValue && this.carbonDimension.getDataType() != DataTypes.STRING) {
--- End diff --
null value updation is also required for String data type second condition in If check is not correct
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184916011
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java ---
@@ -53,6 +55,21 @@ public PrimitiveQueryType(String name, String parentname, int blockIndex,
this.name = name;
this.parentname = parentname;
this.isDirectDictionary = isDirectDictionary;
+ this.isDictionary = true;
+ }
+
+
+ public PrimitiveQueryType(String name, String parentname, int blockIndex,
--- End diff --
One more extra constructor is not required. If dictionary is not null then isDictionary=true
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4638/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r185217366
--- Diff: integration/spark2/src/main/scala/org/apache/spark/util/SparkTypeConverter.scala ---
@@ -97,16 +97,30 @@ private[spark] object SparkTypeConverter {
def getStructChildren(table: CarbonTable, dimName: String): String = {
table.getChildren(dimName).asScala.map(childDim => {
childDim.getDataType.getName.toLowerCase match {
- case "array" => s"${
+ case "array" => if (table.isTransactionalTable) {s"${
--- End diff --
Written the comment of why transactional and no transactional behaviour will be different.
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184963324
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
@@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
for (int i = 0; i < dataLength; i++) {
children.parseAndBitPack(byteArrayInput, dataOutputStream, generator);
}
+ }
+
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
+ throws IOException, KeyGenException {
+ int dataLength = byteArrayInput.getInt();
+ startOffset += Integer.SIZE / Byte.SIZE;
+ dataOutputStream.writeInt(dataLength);
+ if (children instanceof PrimitiveDataType) {
+ if (children.getIsColumnDictionary()) {
+ dataOutputStream.writeInt(generator[children.getSurrogateIndex()].getKeySizeInBytes());
+ startOffset += Integer.SIZE / Byte.SIZE;
--- End diff --
Done
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184977270
--- Diff: processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java ---
@@ -267,14 +267,24 @@ private static String getComplexTypeString(DataField[] dataFields) {
return dimString.toString();
}
+ private static String isDictionaryType(CarbonDimension dimension) {
--- End diff --
This is used to append in a String field.
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184919620
--- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/CarbonDataLoadConfiguration.java ---
@@ -348,6 +356,33 @@ public void setCardinalityFinder(DictionaryCardinalityFinder cardinalityFinder)
return complexKeyGenerators;
}
+ public Boolean[][] createComplexDictionaryFieldIdentification() {
--- End diff --
If not required, please remove this method
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918753
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
@@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
for (int i = 0; i < dataLength; i++) {
children.parseAndBitPack(byteArrayInput, dataOutputStream, generator);
}
+ }
+
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
+ throws IOException, KeyGenException {
+ int dataLength = byteArrayInput.getInt();
+ startOffset += Integer.SIZE / Byte.SIZE;
+ dataOutputStream.writeInt(dataLength);
+ if (children instanceof PrimitiveDataType) {
+ if (children.getIsColumnDictionary()) {
+ dataOutputStream.writeInt(generator[children.getSurrogateIndex()].getKeySizeInBytes());
+ startOffset += Integer.SIZE / Byte.SIZE;
--- End diff --
No need to calculate every time , just use 4 and add comment
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184928085
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/column/ColumnSchema.java ---
@@ -43,6 +43,9 @@
*/
private static final long serialVersionUID = 7676766554874863763L;
+ public void columnSchema() {
--- End diff --
Removed. No Use.
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184963740
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java ---
@@ -80,7 +82,13 @@ void writeByteArray(T input, DataOutputStream dataOutputStream)
* @throws KeyGenException
*/
void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
- KeyGenerator[] generator) throws IOException, KeyGenException;
+ KeyGenerator[] generator)
+ throws IOException, KeyGenException;
+
+
+ int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
+ throws IOException, KeyGenException;
--- End diff --
complexDictionaryIndentification is removed but StartOffset is used.
---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4620/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184961740
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java ---
@@ -108,4 +128,14 @@ public PrimitiveQueryType(String name, String parentname, int blockIndex,
}
return actualData;
}
+
+ @Override public Object getDataBasedOnDataTypeFromNoDictionary(ByteBuffer data) {
--- End diff --
Done
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4358/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184976629
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java ---
@@ -71,6 +71,8 @@ void writeByteArray(T input, DataOutputStream dataOutputStream)
*/
void setSurrogateIndex(int surrIndex);
+ Boolean getIsColumnDictionary();
--- End diff --
Done
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184919569
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
dataOutputStream.write(v);
}
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
--- End diff --
What is the use of `complexDictionaryIndentification` if not used
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4403/
---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4138/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184962372
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
@@ -171,9 +175,9 @@ public void fillCardinality(List<Integer> dimCardWithComplex) {
/**
* parse byte array and bit pack
*/
- @Override
- public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
- KeyGenerator[] generator) throws IOException, KeyGenException {
+ @Override public void parseAndBitPack(ByteBuffer byteArrayInput,
--- End diff --
Removed
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5560/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184963673
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java ---
@@ -71,6 +71,8 @@ void writeByteArray(T input, DataOutputStream dataOutputStream)
*/
void setSurrogateIndex(int surrIndex);
+ Boolean getIsColumnDictionary();
--- End diff --
Done
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184976699
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java ---
@@ -53,6 +55,21 @@ public PrimitiveQueryType(String name, String parentname, int blockIndex,
this.name = name;
this.parentname = parentname;
this.isDirectDictionary = isDirectDictionary;
+ this.isDictionary = true;
+ }
+
+
+ public PrimitiveQueryType(String name, String parentname, int blockIndex,
+ org.apache.carbondata.core.metadata.datatype.DataType dataType, int keySize,
+ Dictionary dictionary, boolean isDirectDictionary, boolean isDictionary) {
--- End diff --
Done.
---
[GitHub] carbondata pull request #2209: [WIP][Non Transactional Table]Avro Record Com...
Posted by ajantha-bhat <gi...@git.apache.org>.
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184904922
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestNonTransactionalCarbonTable.scala ---
@@ -175,9 +175,31 @@ class TestNonTransactionalCarbonTable extends QueryTest with BeforeAndAfterAll {
test("test create External Table with Schema with partition, should ignore schema and partition")
{
- buildTestDataSingleFile()
+ sql("DROP TABLE IF EXISTS sdkOutputTable")
+
+ // with partition
+ sql("CREATE EXTERNAL TABLE sdkOutputTable(name string) PARTITIONED BY (age int) STORED BY 'carbondata' LOCATION '/home/root1/avro/files' ")
+//
+// checkAnswer(sql("select * from sdkOutputTable"), Seq(Row("robot0", 0, 0.0),
+// Row("robot1", 1, 0.5),
+// Row("robot2", 2, 1.0)))
--- End diff --
please revert back this test case and add new one
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184917500
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java ---
@@ -84,6 +101,9 @@ public PrimitiveQueryType(String name, String parentname, int blockIndex,
DimensionRawColumnChunk[] rawColumnChunks, int rowNumber,
int pageNumber, DataOutputStream dataOutputStream) throws IOException {
byte[] currentVal = copyBlockDataChunk(rawColumnChunks, rowNumber, pageNumber);
+ if (!this.isDictionary) {
+ dataOutputStream.writeInt(currentVal.length);
--- End diff --
Just writing length? where are you writing data?
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184919435
--- Diff: processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java ---
@@ -267,14 +267,24 @@ private static String getComplexTypeString(DataField[] dataFields) {
return dimString.toString();
}
+ private static String isDictionaryType(CarbonDimension dimension) {
--- End diff --
why this method return type is string??? Change this to string
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4623/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184964754
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -94,6 +99,11 @@
private CarbonDimension carbonDimension;
+ private Boolean isDictionary;
+
+ private FieldConverter fieldConverterForNoDictionary;
--- End diff --
Removed
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4637/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184962411
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
@@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
for (int i = 0; i < dataLength; i++) {
children.parseAndBitPack(byteArrayInput, dataOutputStream, generator);
}
+ }
+
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
--- End diff --
Removed
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184916986
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java ---
@@ -71,6 +71,8 @@ void writeByteArray(T input, DataOutputStream dataOutputStream)
*/
void setSurrogateIndex(int surrIndex);
+ Boolean getIsColumnDictionary();
--- End diff --
change this to boolean isDictionaryEncoded()
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184917127
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java ---
@@ -53,6 +55,21 @@ public PrimitiveQueryType(String name, String parentname, int blockIndex,
this.name = name;
this.parentname = parentname;
this.isDirectDictionary = isDirectDictionary;
+ this.isDictionary = true;
+ }
+
+
+ public PrimitiveQueryType(String name, String parentname, int blockIndex,
+ org.apache.carbondata.core.metadata.datatype.DataType dataType, int keySize,
+ Dictionary dictionary, boolean isDirectDictionary, boolean isDictionary) {
--- End diff --
I don't think it is required to have a separate constructor, if (dictionary == null && !isDirectDictionary) then it becomes nodictionary
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184917041
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -94,6 +99,11 @@
private CarbonDimension carbonDimension;
+ private Boolean isDictionary;
--- End diff --
change this to boolean isDictionary
---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5518/
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4661/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184976649
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -94,6 +99,11 @@
private CarbonDimension carbonDimension;
+ private Boolean isDictionary;
--- End diff --
Done
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184976884
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java ---
@@ -84,6 +101,9 @@ public PrimitiveQueryType(String name, String parentname, int blockIndex,
DimensionRawColumnChunk[] rawColumnChunks, int rowNumber,
int pageNumber, DataOutputStream dataOutputStream) throws IOException {
byte[] currentVal = copyBlockDataChunk(rawColumnChunks, rowNumber, pageNumber);
+ if (!this.isDictionary) {
+ dataOutputStream.writeInt(currentVal.length);
--- End diff --
Below line 105 we are writing the data. Data writing is without any condition.
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184928356
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/collector/impl/DictionaryBasedResultCollector.java ---
@@ -134,7 +134,13 @@ void fillDimensionData(BlockletScannedResult scannedResult, int[] surrogateResul
row[order[i]] =
DataTypeUtil.getDataBasedOnDataType(scannedResult.getBlockletId(), DataTypes.STRING);
}
- } else {
+ } else if (complexDataTypeArray[i]) {
+ // Complex Type With No Dictionary Encoding.
+ row[order[i]] = comlexDimensionInfoMap.get(queryDimensions[i].getDimension().getOrdinal())
+ .getDataBasedOnDataTypeFromNoDictionary(
+ ByteBuffer.wrap(complexTypeKeyArray[complexTypeColumnIndex++]));
+ } else
+ {
--- End diff --
Done
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4395/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918548
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
@@ -171,9 +175,9 @@ public void fillCardinality(List<Integer> dimCardWithComplex) {
/**
* parse byte array and bit pack
*/
- @Override
- public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
- KeyGenerator[] generator) throws IOException, KeyGenException {
+ @Override public void parseAndBitPack(ByteBuffer byteArrayInput,
--- End diff --
Now this method is not used? if not delete it
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
@sounakr Please fix the build
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184927388
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java ---
@@ -53,6 +55,21 @@ public PrimitiveQueryType(String name, String parentname, int blockIndex,
this.name = name;
this.parentname = parentname;
this.isDirectDictionary = isDirectDictionary;
+ this.isDictionary = true;
+ }
+
+
+ public PrimitiveQueryType(String name, String parentname, int blockIndex,
--- End diff --
Done. But we need the parameter as dictionary field is always set.
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184968935
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
dataOutputStream.write(v);
}
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
--- End diff --
Removed
---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4355/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r185213084
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -211,29 +241,66 @@ public int getSurrogateIndex() {
/*
* set surrogate index
*/
- @Override
- public void setSurrogateIndex(int surrIndex) {
- index = surrIndex;
+ @Override public void setSurrogateIndex(int surrIndex) {
+ if (this.carbonDimension != null && !this.carbonDimension.hasEncoding(Encoding.DICTIONARY)) {
+ index = 0;
+ } else if (this.carbonDimension == null && isDictionary == false) {
+ index = 0;
+ } else {
+ index = surrIndex;
+ }
+ }
+
+ @Override public Boolean getIsColumnDictionary() {
+ return isDictionary;
}
@Override public void writeByteArray(Object input, DataOutputStream dataOutputStream)
throws IOException, DictionaryGenerationException {
String parsedValue =
input == null ? null : DataTypeUtil.parseValue(input.toString(), carbonDimension);
- Integer surrogateKey;
- if (null == parsedValue) {
- surrogateKey = CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
- } else {
- surrogateKey = dictionaryGenerator.getOrGenerateKey(parsedValue);
- if (surrogateKey == CarbonCommonConstants.INVALID_SURROGATE_KEY) {
+ if (this.isDictionary) {
+ Integer surrogateKey;
+ if (null == parsedValue) {
surrogateKey = CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
+ } else {
+ surrogateKey = dictionaryGenerator.getOrGenerateKey(parsedValue);
+ if (surrogateKey == CarbonCommonConstants.INVALID_SURROGATE_KEY) {
+ surrogateKey = CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
+ }
+ }
+ dataOutputStream.writeInt(surrogateKey);
+ } else {
+ // Transform into ByteArray for No Dictionary.
+ // TODO have to refactor and place all the cases present in NonDictionaryFieldConverterImpl
+ if (null == parsedValue && this.carbonDimension.getDataType() != DataTypes.STRING) {
--- End diff --
The check has to be similar to NonDictionaryFieldConverterImpl
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918720
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
@@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
for (int i = 0; i < dataLength; i++) {
children.parseAndBitPack(byteArrayInput, dataOutputStream, generator);
}
+ }
+
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
+ throws IOException, KeyGenException {
+ int dataLength = byteArrayInput.getInt();
+ startOffset += Integer.SIZE / Byte.SIZE;
--- End diff --
No need to calculate every time , just use `4` and add comment
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184976246
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
@@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
for (int i = 0; i < dataLength; i++) {
children.parseAndBitPack(byteArrayInput, dataOutputStream, generator);
}
+ }
+
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
+ throws IOException, KeyGenException {
+ int dataLength = byteArrayInput.getInt();
+ startOffset += Integer.SIZE / Byte.SIZE;
--- End diff --
Added CarbonCommonConstant
---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5510/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184919901
--- Diff: processing/src/main/java/org/apache/carbondata/processing/store/TablePage.java ---
@@ -222,8 +222,9 @@ private void addComplexColumn(int index, int rowId, byte[] complexColumns) {
ByteBuffer byteArrayInput = ByteBuffer.wrap(complexColumns);
ByteArrayOutputStream byteArrayOutput = new ByteArrayOutputStream();
DataOutputStream dataOutputStream = new DataOutputStream(byteArrayOutput);
- complexDataType.parseAndBitPack(byteArrayInput, dataOutputStream,
- model.getComplexDimensionKeyGenerator());
+ int startOffset = 0;
--- End diff --
What is the use of `startOffset` when not used?
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184956997
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java ---
@@ -53,6 +55,21 @@ public PrimitiveQueryType(String name, String parentname, int blockIndex,
this.name = name;
this.parentname = parentname;
this.isDirectDictionary = isDirectDictionary;
+ this.isDictionary = true;
+ }
+
+
+ public PrimitiveQueryType(String name, String parentname, int blockIndex,
+ org.apache.carbondata.core.metadata.datatype.DataType dataType, int keySize,
+ Dictionary dictionary, boolean isDirectDictionary, boolean isDictionary) {
--- End diff --
Done
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184916825
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
@@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
for (int i = 0; i < dataLength; i++) {
children.parseAndBitPack(byteArrayInput, dataOutputStream, generator);
}
+ }
+
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
--- End diff --
complexDictionaryIndentification is not required as each primitive children knows it is not of dictionary type or no dictionary type
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184962348
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
@@ -171,9 +175,9 @@ public void fillCardinality(List<Integer> dimCardWithComplex) {
/**
* parse byte array and bit pack
*/
- @Override
- public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
- KeyGenerator[] generator) throws IOException, KeyGenException {
+ @Override public void parseAndBitPack(ByteBuffer byteArrayInput,
--- End diff --
Done
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184968977
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
dataOutputStream.write(v);
}
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
--- End diff --
Currently no required. Removed
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184976034
--- Diff: processing/src/main/java/org/apache/carbondata/processing/store/CarbonFactDataHandlerModel.java ---
@@ -609,5 +616,13 @@ public short getWritingCoresCount() {
public DataMapWriterListener getDataMapWriterlistener() {
return dataMapWriterlistener;
}
+
+ public Boolean[][] getComplexDictionaryFields() {
--- End diff --
Done
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4654/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184992863
--- Diff: processing/src/main/java/org/apache/carbondata/processing/store/TablePage.java ---
@@ -222,8 +222,9 @@ private void addComplexColumn(int index, int rowId, byte[] complexColumns) {
ByteBuffer byteArrayInput = ByteBuffer.wrap(complexColumns);
ByteArrayOutputStream byteArrayOutput = new ByteArrayOutputStream();
DataOutputStream dataOutputStream = new DataOutputStream(byteArrayOutput);
- complexDataType.parseAndBitPack(byteArrayInput, dataOutputStream,
- model.getComplexDimensionKeyGenerator());
+ int startOffset = 0;
--- End diff --
Removed
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184969127
--- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/CarbonDataLoadConfiguration.java ---
@@ -348,6 +356,33 @@ public void setCardinalityFinder(DictionaryCardinalityFinder cardinalityFinder)
return complexKeyGenerators;
}
+ public Boolean[][] createComplexDictionaryFieldIdentification() {
--- End diff --
removed
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4624/
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5558/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918803
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
dataOutputStream.write(v);
}
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
--- End diff --
Boolean[][] complexDictionaryIndentification is not required as each Primitive type field knows whether it is of dictionary/ no dictionary type.
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184961725
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestNonTransactionalCarbonTable.scala ---
@@ -175,9 +175,31 @@ class TestNonTransactionalCarbonTable extends QueryTest with BeforeAndAfterAll {
test("test create External Table with Schema with partition, should ignore schema and partition")
{
- buildTestDataSingleFile()
+ sql("DROP TABLE IF EXISTS sdkOutputTable")
+
+ // with partition
+ sql("CREATE EXTERNAL TABLE sdkOutputTable(name string) PARTITIONED BY (age int) STORED BY 'carbondata' LOCATION '/home/root1/avro/files' ")
+//
+// checkAnswer(sql("select * from sdkOutputTable"), Seq(Row("robot0", 0, 0.0),
+// Row("robot1", 1, 0.5),
+// Row("robot2", 2, 1.0)))
--- End diff --
Done
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4622/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184929215
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java ---
@@ -53,6 +55,21 @@ public PrimitiveQueryType(String name, String parentname, int blockIndex,
this.name = name;
this.parentname = parentname;
this.isDirectDictionary = isDirectDictionary;
+ this.isDictionary = true;
+ }
+
+
+ public PrimitiveQueryType(String name, String parentname, int blockIndex,
--- End diff --
Moved into a single constructor. But we cannot check is the column is a dictionary or nodictionary through the existing variable dictionary as it is always going to get filled up by default.
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184920004
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java ---
@@ -80,7 +82,13 @@ void writeByteArray(T input, DataOutputStream dataOutputStream)
* @throws KeyGenException
*/
void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
- KeyGenerator[] generator) throws IOException, KeyGenException;
+ KeyGenerator[] generator)
+ throws IOException, KeyGenException;
+
+
+ int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
+ throws IOException, KeyGenException;
--- End diff --
Remove `complexDictionaryIndentification` and `startOffset` if not used
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184918157
--- Diff: integration/spark2/src/main/scala/org/apache/spark/util/SparkTypeConverter.scala ---
@@ -97,16 +97,30 @@ private[spark] object SparkTypeConverter {
def getStructChildren(table: CarbonTable, dimName: String): String = {
table.getChildren(dimName).asScala.map(childDim => {
childDim.getDataType.getName.toLowerCase match {
- case "array" => s"${
+ case "array" => if (table.isTransactionalTable) {s"${
--- End diff --
Could explain in comments why this many checks are required for non transactional table?
Better write another method rather than keeping so many if checks
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184917833
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java ---
@@ -108,4 +128,14 @@ public PrimitiveQueryType(String name, String parentname, int blockIndex,
}
return actualData;
}
+
+ @Override public Object getDataBasedOnDataTypeFromNoDictionary(ByteBuffer data) {
--- End diff --
I don't think it is required to have one more method, just rename and handle in same method `getDataBasedOnDataTypeFromSurrogates` , it can avoid duplicate code and extra handling
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184976193
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
@@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
for (int i = 0; i < dataLength; i++) {
children.parseAndBitPack(byteArrayInput, dataOutputStream, generator);
}
+ }
+
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
+ throws IOException, KeyGenException {
+ int dataLength = byteArrayInput.getInt();
+ startOffset += Integer.SIZE / Byte.SIZE;
+ dataOutputStream.writeInt(dataLength);
+ if (children instanceof PrimitiveDataType) {
+ if (children.getIsColumnDictionary()) {
+ dataOutputStream.writeInt(generator[children.getSurrogateIndex()].getKeySizeInBytes());
+ startOffset += Integer.SIZE / Byte.SIZE;
--- End diff --
Added a constant
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4659/
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5565/
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5563/
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4379/
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
@sounakr Please add testcases for complextype also not only in AVRO.
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184916594
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
@@ -171,9 +175,9 @@ public void fillCardinality(List<Integer> dimCardWithComplex) {
/**
* parse byte array and bit pack
*/
- @Override
- public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
- KeyGenerator[] generator) throws IOException, KeyGenException {
+ @Override public void parseAndBitPack(ByteBuffer byteArrayInput,
--- End diff --
Please remove this method from interface as below new method will be used
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184919308
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
dataOutputStream.write(v);
}
+ @Override
+ public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+ KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
+ throws IOException, KeyGenException {
+ if (!this.isDictionary) {
+ int sizeOfData = byteArrayInput.getInt();
+ startOffset += Integer.SIZE / Byte.SIZE;
+ dataOutputStream.writeInt(sizeOfData);
+ byte[] bb = new byte[sizeOfData];
+ byteArrayInput.get(bb, 0, sizeOfData);
+ dataOutputStream.write(bb);
+ startOffset += sizeOfData;
+ } else {
+ int data = byteArrayInput.getInt();
+ startOffset += Integer.SIZE / Byte.SIZE;
--- End diff --
I think this is not required, if it is only writeoffset
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184917713
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
@@ -94,6 +99,11 @@
private CarbonDimension carbonDimension;
+ private Boolean isDictionary;
+
+ private FieldConverter fieldConverterForNoDictionary;
--- End diff --
FieldConverter is not required in this class, as caller will create for primitive type like existing flow..same way we can handle for no dictionary
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184916437
--- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
@@ -171,9 +175,9 @@ public void fillCardinality(List<Integer> dimCardWithComplex) {
/**
* parse byte array and bit pack
*/
- @Override
- public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
- KeyGenerator[] generator) throws IOException, KeyGenException {
+ @Override public void parseAndBitPack(ByteBuffer byteArrayInput,
+ DataOutputStream dataOutputStream, KeyGenerator[] generator)
+ throws IOException, KeyGenException {
int dataLength = byteArrayInput.getInt();
--- End diff --
remove extra space
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184927567
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/SchemaReader.java ---
@@ -81,10 +81,13 @@ public static TableInfo getTableInfo(AbsoluteTableIdentifier identifier)
}
public static TableInfo inferSchema(AbsoluteTableIdentifier identifier,
- boolean isCarbonFileProvider) throws IOException {
+ boolean isCarbonFileProvider, TableInfo tableInfoFromCache) throws IOException {
// This routine is going to infer schema from the carbondata file footer
// Convert the ColumnSchema -> TableSchema -> TableInfo.
// Return the TableInfo.
+ if (tableInfoFromCache != null) {
--- End diff --
Done
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4401/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ajantha-bhat <gi...@git.apache.org>.
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184905554
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/SchemaReader.java ---
@@ -81,10 +81,13 @@ public static TableInfo getTableInfo(AbsoluteTableIdentifier identifier)
}
public static TableInfo inferSchema(AbsoluteTableIdentifier identifier,
- boolean isCarbonFileProvider) throws IOException {
+ boolean isCarbonFileProvider, TableInfo tableInfoFromCache) throws IOException {
// This routine is going to infer schema from the carbondata file footer
// Convert the ColumnSchema -> TableSchema -> TableInfo.
// Return the TableInfo.
+ if (tableInfoFromCache != null) {
--- End diff --
If tableInfoFromCache is found, No need to call inferschema itself, please handle this outside infer schema
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2209
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4656/
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184916731
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/collector/impl/DictionaryBasedResultCollector.java ---
@@ -134,7 +134,13 @@ void fillDimensionData(BlockletScannedResult scannedResult, int[] surrogateResul
row[order[i]] =
DataTypeUtil.getDataBasedOnDataType(scannedResult.getBlockletId(), DataTypes.STRING);
}
- } else {
+ } else if (complexDataTypeArray[i]) {
+ // Complex Type With No Dictionary Encoding.
+ row[order[i]] = comlexDimensionInfoMap.get(queryDimensions[i].getDimension().getOrdinal())
+ .getDataBasedOnDataTypeFromNoDictionary(
+ ByteBuffer.wrap(complexTypeKeyArray[complexTypeColumnIndex++]));
+ } else
+ {
--- End diff --
Move the braces up
---
[GitHub] carbondata pull request #2209: [CARBONDATA-2388][SDK]Avro Record Complex Typ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2209#discussion_r184916441
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/column/ColumnSchema.java ---
@@ -43,6 +43,9 @@
*/
private static final long serialVersionUID = 7676766554874863763L;
+ public void columnSchema() {
--- End diff --
What is the use of this method
---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4345/
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on the issue:
https://github.com/apache/carbondata/pull/2209
Retest this please
---
[GitHub] carbondata issue #2209: [WIP][Non Transactional Table]Avro Record Complex Ty...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5319/
---
[GitHub] carbondata issue #2209: [CARBONDATA-2388][SDK]Avro Record Complex Type Imple...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2209
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5522/
---