You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by GitBox <gi...@apache.org> on 2021/09/29 17:49:05 UTC
[GitHub] [carbondata] ShreelekhyaG opened a new pull request #4228: [WIP] Empty bad record support for complex type
ShreelekhyaG opened a new pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228
### Why is this PR needed?
1. `IS_EMPTY_DATA_BAD_RECORD` property not supported for complex types.
2. To update documentation that `COLUMN_META_CACHE` and `RANGE_COLUMN `doesn't support complex datatype
### What changes were proposed in this PR?
1. Made changes to pass down `IS_EMPTY_DATA_BAD_RECORD` property and throw exception.
2. Updated document and added testcase.
### Does this PR introduce any user interface change?
- No
### Is any new testcase added?
- Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-933241290
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5989/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-946729852
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6060/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-948593490
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6079/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-948723748
merged to master. closing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-945159402
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/444/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-941188691
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/442/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-938377017
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/423/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-931651897
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/376/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-947513332
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#discussion_r731564786
##########
File path: processing/src/main/java/org/apache/carbondata/processing/loading/converter/impl/FieldEncoderFactory.java
##########
@@ -134,10 +135,10 @@ private BinaryDecoder getBinaryDecoder(String binaryDecoder) {
/**
* Create parser for the carbon column.
*/
- public static GenericDataType createComplexDataType(
- DataField dataField, String nullFormat, BinaryDecoder binaryDecoder) {
- return createComplexType(
- dataField.getColumn(), dataField.getColumn().getColName(), nullFormat, binaryDecoder);
+ public static GenericDataType createComplexDataType(DataField dataField, String nullFormat,
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-931648997
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5977/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-930497425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-938382440
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4288/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-945407815
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#discussion_r730834284
##########
File path: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java
##########
@@ -237,21 +237,31 @@ public boolean getIsColumnDictionary() {
@Override
public void writeByteArray(Object input, DataOutputStream dataOutputStream,
- BadRecordLogHolder logHolder, Boolean isWithoutConverter) throws IOException {
+ BadRecordLogHolder logHolder, Boolean isWithoutConverter, boolean isEmptyBadRecord)
+ throws IOException {
String parsedValue = null;
// write null value
if (null == input || ((this.carbonDimension.getDataType() == DataTypes.STRING
|| this.carbonDimension.getDataType() == DataTypes.VARCHAR) && input.equals(nullFormat))) {
updateNullValue(dataOutputStream, logHolder);
return;
}
+ if (input.equals("") && isEmptyBadRecord) {
+ CarbonBadRecordUtil.setErrorMessage(logHolder, carbonDimension.getColName(),
Review comment:
this can be moved inside updateNullValue method and can be reused
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-945375163
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6055/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-945560941
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4314/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-945576535
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/447/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-946753062
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/450/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-945383321
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/445/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-948391575
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4334/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-934217959
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5997/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-931649771
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4231/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-930522147
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5965/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-944360167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#discussion_r723096064
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/badrecordloger/BadRecordEmptyDataTest.scala
##########
@@ -101,6 +101,47 @@ class BadRecordEmptyDataTest extends QueryTest with BeforeAndAfterAll {
}
}
+ test("Test complex type with empty values and IS_EMPTY_DATA_BAD_RECORD property") {
+ sql("DROP TABLE IF EXISTS complexcarbontable")
+ sql("DROP TABLE IF EXISTS complexhivetable")
+ sql(
+ "create table complexcarbontable(deviceInformationId int, channelsId string, ROMSize " +
+ "string, ROMName String, purchasedate string, mobile struct<imei:string, imsi:int>, MAC " +
+ "array<string>, locationinfo array<struct<ActiveAreaId:int, ActiveCountry:string, " +
Review comment:
add column with map datatype
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-934189418
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4251/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-931967615
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/377/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-941327500
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6052/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#discussion_r731681506
##########
File path: processing/src/main/java/org/apache/carbondata/processing/datatypes/StructDataType.java
##########
@@ -173,22 +174,33 @@ public boolean getIsColumnDictionary() {
}
@Override
- public void writeByteArray(StructObject input, DataOutputStream dataOutputStream,
- BadRecordLogHolder logHolder, Boolean isWithoutConverter) throws IOException {
+ public void writeByteArray(Object input, DataOutputStream dataOutputStream,
+ BadRecordLogHolder logHolder, Boolean isWithoutConverter, boolean isEmptyBadRecord)
+ throws IOException {
dataOutputStream.writeShort(children.size());
- if (input == null) {
+ if (input == null || input.equals("")) {
for (int i = 0; i < children.size(); i++) {
- children.get(i).writeByteArray(null, dataOutputStream, logHolder, isWithoutConverter);
+ if (input != null && input.equals("") && (children.get(i) instanceof ArrayDataType)) {
+ // If child column is of array/map type and is empty, no need to iterate.
Review comment:
```suggestion
// If child column is of array type and is empty, no need to iterate.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-946798968
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#discussion_r730744455
##########
File path: integration/spark/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOpName.scala
##########
@@ -761,7 +761,7 @@ class TestStreamingTableOpName extends QueryTest with BeforeAndAfterAll {
// check one row of streaming data
assert(result(0).isNullAt(0))
assert(result(0).getString(1) == "")
- assert(result(0).isNullAt(9))
+ assert(result(0).getStruct(9).isNullAt(1))
Review comment:
The result is same with change in `is_empty_bad_record `property. As it is used to determine if we need to consider the empty record as bad record or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#discussion_r730741761
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/badrecordloger/BadRecordEmptyDataTest.scala
##########
@@ -101,6 +101,47 @@ class BadRecordEmptyDataTest extends QueryTest with BeforeAndAfterAll {
}
}
+ test("Test complex type with empty values and IS_EMPTY_DATA_BAD_RECORD property") {
+ sql("DROP TABLE IF EXISTS complexcarbontable")
+ sql("DROP TABLE IF EXISTS complexhivetable")
+ sql(
+ "create table complexcarbontable(deviceInformationId int, channelsId string, ROMSize " +
+ "string, ROMName String, purchasedate string, mobile struct<imei:string, imsi:int>, MAC " +
+ "array<string>, locationinfo array<struct<ActiveAreaId:int, ActiveCountry:string, " +
+ "ActiveProvince:string, Activecity:string, ActiveDistrict:string, ActiveStreet:string>>, " +
+ "proddate struct<productionDate:string,activeDeactivedate:array<string>>, gamePointId " +
+ "double,contractNumber double) STORED AS carbondata")
+ sql("LOAD DATA local inpath '" + resourcesPath +
+ "/complextypeWithEmptyRecords.csv' INTO table complexcarbontable OPTIONS('DELIMITER'=','," +
+ "'QUOTECHAR'='\"', 'FILEHEADER'='deviceInformationId,channelsId,ROMSize,ROMName," +
+ "purchasedate,mobile,MAC,locationinfo,proddate,gamePointId,contractNumber', " +
+ "'COMPLEX_DELIMITER_LEVEL_1'='$', 'COMPLEX_DELIMITER_LEVEL_2'=':', " +
+ "'bad_records_logger_enable'='true','IS_EMPTY_DATA_BAD_RECORD'='true' ," +
+ "'bad_records_action'='ignore')")
Review comment:
Done
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/badrecordloger/BadRecordEmptyDataTest.scala
##########
@@ -101,6 +101,47 @@ class BadRecordEmptyDataTest extends QueryTest with BeforeAndAfterAll {
}
}
+ test("Test complex type with empty values and IS_EMPTY_DATA_BAD_RECORD property") {
+ sql("DROP TABLE IF EXISTS complexcarbontable")
+ sql("DROP TABLE IF EXISTS complexhivetable")
+ sql(
+ "create table complexcarbontable(deviceInformationId int, channelsId string, ROMSize " +
+ "string, ROMName String, purchasedate string, mobile struct<imei:string, imsi:int>, MAC " +
+ "array<string>, locationinfo array<struct<ActiveAreaId:int, ActiveCountry:string, " +
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-945159634
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6054/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-932209386
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4241/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-932209615
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-941188691
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-941321511
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4308/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-938378272
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6033/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#discussion_r723094923
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/badrecordloger/BadRecordEmptyDataTest.scala
##########
@@ -101,6 +101,47 @@ class BadRecordEmptyDataTest extends QueryTest with BeforeAndAfterAll {
}
}
+ test("Test complex type with empty values and IS_EMPTY_DATA_BAD_RECORD property") {
+ sql("DROP TABLE IF EXISTS complexcarbontable")
+ sql("DROP TABLE IF EXISTS complexhivetable")
+ sql(
+ "create table complexcarbontable(deviceInformationId int, channelsId string, ROMSize " +
+ "string, ROMName String, purchasedate string, mobile struct<imei:string, imsi:int>, MAC " +
+ "array<string>, locationinfo array<struct<ActiveAreaId:int, ActiveCountry:string, " +
+ "ActiveProvince:string, Activecity:string, ActiveDistrict:string, ActiveStreet:string>>, " +
+ "proddate struct<productionDate:string,activeDeactivedate:array<string>>, gamePointId " +
+ "double,contractNumber double) STORED AS carbondata")
+ sql("LOAD DATA local inpath '" + resourcesPath +
+ "/complextypeWithEmptyRecords.csv' INTO table complexcarbontable OPTIONS('DELIMITER'=','," +
+ "'QUOTECHAR'='\"', 'FILEHEADER'='deviceInformationId,channelsId,ROMSize,ROMName," +
+ "purchasedate,mobile,MAC,locationinfo,proddate,gamePointId,contractNumber', " +
+ "'COMPLEX_DELIMITER_LEVEL_1'='$', 'COMPLEX_DELIMITER_LEVEL_2'=':', " +
+ "'bad_records_logger_enable'='true','IS_EMPTY_DATA_BAD_RECORD'='true' ," +
+ "'bad_records_action'='ignore')")
Review comment:
can you add one validation with 'IS_EMPTY_DATA_BAD_RECORD'='true' and 'bad_records_action'='fail'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-945545437
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6057/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#discussion_r730808999
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
##########
@@ -1059,6 +1060,9 @@ object CommonUtil {
val nullFormat = configuration
.getDataLoadProperty(DataLoadProcessorConstants.SERIALIZATION_NULL_FORMAT)
.toString
+ val isEmptyBadRecord = (configuration
Review comment:
variable not used. please revert this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-948503542
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#discussion_r731565065
##########
File path: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java
##########
@@ -237,21 +237,31 @@ public boolean getIsColumnDictionary() {
@Override
public void writeByteArray(Object input, DataOutputStream dataOutputStream,
- BadRecordLogHolder logHolder, Boolean isWithoutConverter) throws IOException {
+ BadRecordLogHolder logHolder, Boolean isWithoutConverter, boolean isEmptyBadRecord)
+ throws IOException {
String parsedValue = null;
// write null value
if (null == input || ((this.carbonDimension.getDataType() == DataTypes.STRING
|| this.carbonDimension.getDataType() == DataTypes.VARCHAR) && input.equals(nullFormat))) {
updateNullValue(dataOutputStream, logHolder);
return;
}
+ if (input.equals("") && isEmptyBadRecord) {
+ CarbonBadRecordUtil.setErrorMessage(logHolder, carbonDimension.getColName(),
Review comment:
Done. Used new method updateEmptyValue
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/badrecordloger/BadRecordEmptyDataTest.scala
##########
@@ -101,6 +101,60 @@ class BadRecordEmptyDataTest extends QueryTest with BeforeAndAfterAll {
}
}
+ test("Test complex type with empty values and IS_EMPTY_DATA_BAD_RECORD property") {
+ sql("DROP TABLE IF EXISTS complexcarbontable")
+ sql("DROP TABLE IF EXISTS complexhivetable")
+ sql(
+ "create table complexcarbontable(deviceInformationId int, channelsId string, ROMSize " +
+ "string, ROMName String, purchasedate string, file struct<school:array<string>, age:int>," +
+ " MAC map<string, int>, locationinfo array<struct<ActiveAreaId:int, ActiveCountry:string, " +
+ "ActiveProvince:string, Activecity:string, ActiveDistrict:string, ActiveStreet:string>>, " +
+ "proddate struct<productionDate:string,activeDeactivedate:array<string>>, gamePointId " +
+ "double,contractNumber double, st struct<school:struct<a:string,b:int>, age:int>," +
+ "ar array<array<string>>) STORED AS carbondata")
+ val exception = intercept[Exception] (sql("LOAD DATA local inpath '" + resourcesPath +
+ "/complextypeWithEmptyRecords.csv' INTO table complexcarbontable OPTIONS('DELIMITER'=','," +
+ "'QUOTECHAR'='\"', 'FILEHEADER'='deviceInformationId,channelsId,ROMSize,ROMName," +
+ "purchasedate,file,MAC,locationinfo,proddate,gamePointId,contractNumber,st,ar', " +
+ "'COMPLEX_DELIMITER_LEVEL_1'='$', 'COMPLEX_DELIMITER_LEVEL_2'=':', " +
+ "'bad_records_logger_enable'='true','IS_EMPTY_DATA_BAD_RECORD'='true' ," +
+ "'bad_records_action'='fail')"))
+ assert(exception.getMessage.contains(
+ "The value with column name file.age and column data type INT is not a valid INT type."))
+ sql("LOAD DATA local inpath '" + resourcesPath +
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-948395407
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6077/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#discussion_r731904757
##########
File path: processing/src/main/java/org/apache/carbondata/processing/datatypes/StructDataType.java
##########
@@ -173,22 +174,33 @@ public boolean getIsColumnDictionary() {
}
@Override
- public void writeByteArray(StructObject input, DataOutputStream dataOutputStream,
- BadRecordLogHolder logHolder, Boolean isWithoutConverter) throws IOException {
+ public void writeByteArray(Object input, DataOutputStream dataOutputStream,
+ BadRecordLogHolder logHolder, Boolean isWithoutConverter, boolean isEmptyBadRecord)
+ throws IOException {
dataOutputStream.writeShort(children.size());
- if (input == null) {
+ if (input == null || input.equals("")) {
for (int i = 0; i < children.size(); i++) {
- children.get(i).writeByteArray(null, dataOutputStream, logHolder, isWithoutConverter);
+ if (input != null && input.equals("") && (children.get(i) instanceof ArrayDataType)) {
+ // If child column is of array/map type and is empty, no need to iterate.
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-946927163
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6061/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-931907190
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4232/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-930497425
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/365/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-946433437
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/448/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-946777087
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4317/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-946956948
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/451/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#discussion_r733315196
##########
File path: processing/src/main/java/org/apache/carbondata/processing/loading/steps/InputProcessorStepWithNoConverterImpl.java
##########
@@ -443,8 +443,11 @@ private void getComplexTypeByteArray(Object[] newData, int index, Object[] data,
try {
GenericDataType complextType =
dataFieldsWithComplexDataType.get(dataField.getColumn().getOrdinal());
- complextType
- .writeByteArray(data[orderedIndex], dataOutputStream, logHolder, isWithoutConverter);
+ boolean isEmptyBadRecord = Boolean.parseBoolean(
+ configuration.getDataLoadProperty(DataLoadProcessorConstants.IS_EMPTY_DATA_BAD_RECORD)
+ .toString());
+ complextType.writeByteArray(data[orderedIndex], dataOutputStream, logHolder,
Review comment:
```suggestion
complexType.writeByteArray(data[orderedIndex], dataOutputStream, logHolder,
```
##########
File path: processing/src/main/java/org/apache/carbondata/processing/util/CarbonBadRecordUtil.java
##########
@@ -154,4 +158,23 @@ public static String getBadRecordsPath(Map<String, String> loadOptions, CarbonTa
return badRecordsPath;
}
+ public static void updateEmptyValue(DataOutputStream dataOutputStream, boolean isEmptyBadRecord,
+ BadRecordLogHolder logHolder, String parentName, DataType dataType) throws IOException {
+ CarbonUtil.updateWithEmptyValueBasedOnDatatype(dataOutputStream, dataType);
+ if (isEmptyBadRecord) {
+ CarbonBadRecordUtil.setErrorMessage(logHolder, parentName, dataType.getName());
+ }
+ return;
Review comment:
this statement is not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-946418057
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4315/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-939799393
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/435/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-938166783
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/422/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-945159316
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4311/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-938172200
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4287/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-934118260
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-931903358
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5978/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-948503402
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-947551289
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6069/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-947622445
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4326/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-947649585
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/459/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-948419260
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/467/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#discussion_r730837875
##########
File path: processing/src/main/java/org/apache/carbondata/processing/loading/converter/impl/FieldEncoderFactory.java
##########
@@ -134,10 +135,10 @@ private BinaryDecoder getBinaryDecoder(String binaryDecoder) {
/**
* Create parser for the carbon column.
*/
- public static GenericDataType createComplexDataType(
- DataField dataField, String nullFormat, BinaryDecoder binaryDecoder) {
- return createComplexType(
- dataField.getColumn(), dataField.getColumn().getColName(), nullFormat, binaryDecoder);
+ public static GenericDataType createComplexDataType(DataField dataField, String nullFormat,
Review comment:
please revert this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-930505714
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4220/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-933231687
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4243/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-938166783
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-939791038
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6045/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-939798559
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4301/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-938532788
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4289/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-934209613
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/396/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ydvpankaj99 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
ydvpankaj99 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-933150900
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-948597956
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4337/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#discussion_r730811274
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/badrecordloger/BadRecordEmptyDataTest.scala
##########
@@ -101,6 +101,60 @@ class BadRecordEmptyDataTest extends QueryTest with BeforeAndAfterAll {
}
}
+ test("Test complex type with empty values and IS_EMPTY_DATA_BAD_RECORD property") {
+ sql("DROP TABLE IF EXISTS complexcarbontable")
+ sql("DROP TABLE IF EXISTS complexhivetable")
+ sql(
+ "create table complexcarbontable(deviceInformationId int, channelsId string, ROMSize " +
+ "string, ROMName String, purchasedate string, file struct<school:array<string>, age:int>," +
+ " MAC map<string, int>, locationinfo array<struct<ActiveAreaId:int, ActiveCountry:string, " +
+ "ActiveProvince:string, Activecity:string, ActiveDistrict:string, ActiveStreet:string>>, " +
+ "proddate struct<productionDate:string,activeDeactivedate:array<string>>, gamePointId " +
+ "double,contractNumber double, st struct<school:struct<a:string,b:int>, age:int>," +
+ "ar array<array<string>>) STORED AS carbondata")
+ val exception = intercept[Exception] (sql("LOAD DATA local inpath '" + resourcesPath +
+ "/complextypeWithEmptyRecords.csv' INTO table complexcarbontable OPTIONS('DELIMITER'=','," +
+ "'QUOTECHAR'='\"', 'FILEHEADER'='deviceInformationId,channelsId,ROMSize,ROMName," +
+ "purchasedate,file,MAC,locationinfo,proddate,gamePointId,contractNumber,st,ar', " +
+ "'COMPLEX_DELIMITER_LEVEL_1'='$', 'COMPLEX_DELIMITER_LEVEL_2'=':', " +
+ "'bad_records_logger_enable'='true','IS_EMPTY_DATA_BAD_RECORD'='true' ," +
+ "'bad_records_action'='fail')"))
+ assert(exception.getMessage.contains(
+ "The value with column name file.age and column data type INT is not a valid INT type."))
+ sql("LOAD DATA local inpath '" + resourcesPath +
Review comment:
can move this to local method and reuse
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-945366616
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4312/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-945157102
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG closed pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG closed pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-946931562
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4318/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-948664868
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/470/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-946415092
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6058/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-938549473
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6034/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-938539518
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/424/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-938172395
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6032/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4228: [CARBONDATA-4298][CARBONDATA-4281] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#discussion_r723109539
##########
File path: integration/spark/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOpName.scala
##########
@@ -761,7 +761,7 @@ class TestStreamingTableOpName extends QueryTest with BeforeAndAfterAll {
// check one row of streaming data
assert(result(0).isNullAt(0))
assert(result(0).getString(1) == "")
- assert(result(0).isNullAt(9))
+ assert(result(0).getStruct(9).isNullAt(1))
Review comment:
can add is_empty_bad_record property = true, instead of changing the test case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4228: [WIP] Empty bad record support for complex type
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4228:
URL: https://github.com/apache/carbondata/pull/4228#issuecomment-933235721
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/388/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org