You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by anubhav100 <gi...@git.apache.org> on 2018/02/06 08:21:22 UTC
[GitHub] carbondata pull request #1934: [CARBONDATA-2133] Fixed Exception displays af...
GitHub user anubhav100 opened a pull request:
https://github.com/apache/carbondata/pull/1934
[CARBONDATA-2133] Fixed Exception displays after performing select query on newly added Boolean Type
**Problem** : In Restructure util and RestructureBasedVectorResultCollector to get the default value of a measure type the case for boolean data type was missing,and in DataTypeUtil to store default value in bytes case of boolean data type was missing
**Solution**:Add the Required Cases
**Complete Description**:
**Classes changed**:
1.**RestructureBasedVectorResultCollector**:
**in the below method case for boolean data type was missing**
private void fillDataForNonExistingMeasures() {
for (int i = 0; i < tableBlockExecutionInfos.getActualQueryMeasures().length; i++) {
if (!measureInfo.getMeasureExists()[i]) {
int queryOrder = tableBlockExecutionInfos.getActualQueryMeasures()[i].getQueryOrder();
CarbonMeasure measure = tableBlockExecutionInfos.getActualQueryMeasures()[i].getMeasure();
ColumnVectorInfo columnVectorInfo = allColumnInfo[queryOrder];
CarbonColumnVector vector = columnVectorInfo.vector;
Object defaultValue = measureDefaultValues[i];
if (null == defaultValue) {
vector.putNulls(columnVectorInfo.vectorOffset, columnVectorInfo.size);
} else {
DataType dataType = measureInfo.getMeasureDataTypes()[i];
if (dataType == DataTypes.SHORT) {
vector.putShorts(columnVectorInfo.vectorOffset, columnVectorInfo.size,
(short) defaultValue);
} else if (dataType == DataTypes.INT) {
vector
.putInts(columnVectorInfo.vectorOffset, columnVectorInfo.size, (int) defaultValue);
} else if (dataType == DataTypes.LONG) {
vector.putLongs(columnVectorInfo.vectorOffset, columnVectorInfo.size,
(long) defaultValue);
} else if (DataTypes.isDecimal(dataType)) {
vector.putDecimals(columnVectorInfo.vectorOffset, columnVectorInfo.size,
((Decimal) defaultValue).toJavaBigDecimal(), measure.getPrecision());
} else {
vector.putDoubles(columnVectorInfo.vectorOffset, columnVectorInfo.size,
(double) defaultValue);
}
}
}
}
}
2.**RestructureUtil**:
**in below methods getMeasureDefaultValue,getMeasureDefaultValueByType the if statement for boolean data type was missing**
public static Object getMeasureDefaultValue(ColumnSchema columnSchema, byte[] defaultValue) {
Object measureDefaultValue = null;
if (!isDefaultValueNull(defaultValue)) {
String value;
DataType dataType = columnSchema.getDataType();
if (dataType == DataTypes.SHORT) {
value = new String(defaultValue, Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
measureDefaultValue = Short.valueOf(value);
} else if (dataType == DataTypes.LONG) {
value = new String(defaultValue, Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
measureDefaultValue = Long.parseLong(value);
} else if (dataType == DataTypes.INT) {
value = new String(defaultValue, Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
measureDefaultValue = Integer.parseInt(value);
} else if (DataTypes.isDecimal(dataType)) {
BigDecimal decimal = DataTypeUtil.byteToBigDecimal(defaultValue);
if (columnSchema.getScale() > decimal.scale()) {
decimal = decimal.setScale(columnSchema.getScale(), RoundingMode.HALF_UP);
}
measureDefaultValue = decimal;
} else {
value = new String(defaultValue, Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
Double parsedValue = Double.valueOf(value);
if (!Double.isInfinite(parsedValue) && !Double.isNaN(parsedValue)) {
measureDefaultValue = parsedValue;
}
}
}
return measureDefaultValue;
}
3.**DatatypeUtil**: To store default value on bytes case of boolean data type was missing
public static byte[] convertDataToBytesBasedOnDataType(String data, ColumnSchema columnSchema) {
if (null == data) {
return null;
} else if (CarbonCommonConstants.MEMBER_DEFAULT_VAL.equals(data)) {
LOGGER.error("Default value should not be carbon specific null value : " + data);
return null;
}
try {
long parsedIntVal = 0;
DataType dataType = columnSchema.getDataType();
if (dataType == DataTypes.INT) {
parsedIntVal = (long) Integer.parseInt(data);
return String.valueOf(parsedIntVal)
.getBytes(Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
} else if (dataType == DataTypes.SHORT) {
parsedIntVal = (long) Short.parseShort(data);
return String.valueOf(parsedIntVal)
.getBytes(Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
} else if (dataType == DataTypes.DOUBLE) {
return String.valueOf(Double.parseDouble(data))
.getBytes(Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
} else if (dataType == DataTypes.LONG) {
return String.valueOf(Long.parseLong(data))
.getBytes(Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
} else if (dataType == DataTypes.DATE) {
DirectDictionaryGenerator directDictionaryGenerator = DirectDictionaryKeyGeneratorFactory
.getDirectDictionaryGenerator(columnSchema.getDataType());
int value = directDictionaryGenerator.generateDirectSurrogateKey(data);
return String.valueOf(value)
.getBytes(Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
} else if (dataType == DataTypes.TIMESTAMP) {
if (columnSchema.hasEncoding(Encoding.DIRECT_DICTIONARY)) {
DirectDictionaryGenerator directDictionaryGenerator1 = DirectDictionaryKeyGeneratorFactory
.getDirectDictionaryGenerator(columnSchema.getDataType());
int value1 = directDictionaryGenerator1.generateDirectSurrogateKey(data);
return String.valueOf(value1)
.getBytes(Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
} else {
try {
Date dateToStr = timeStampformatter.get().parse(data);
return ByteUtil.toBytes(dateToStr.getTime());
} catch (ParseException e) {
LOGGER.error(
"Cannot convert value to Time/Long type value. Value is considered as null" + e
.getMessage());
return null;
}
}
} else if (DataTypes.isDecimal(dataType)) {
String parsedValue = parseStringToBigDecimal(data, columnSchema);
if (null == parsedValue) {
return null;
}
java.math.BigDecimal javaDecVal = new java.math.BigDecimal(parsedValue);
return bigDecimalToByte(javaDecVal);
} else {
return getDataTypeConverter().convertFromStringToByte(data);
}
} catch (NumberFormatException ex) {
LOGGER.error("Problem while converting data type" + data);
return null;
}
}
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/anubhav100/incubator-carbondata CARBONDATA-2133
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1934.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1934
----
commit fcd812a9d09302c1d10b2897883fa3c31a86e156
Author: anubhav100 <an...@...>
Date: 2018-02-06T08:03:39Z
Fixed Exception displays after performing select query on newly added Boolean data type
----
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1934
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3589/
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1934
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2502/
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1934
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3543/
---
[GitHub] carbondata pull request #1934: [CARBONDATA-2133] Fixed Exception displays af...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1934#discussion_r168088520
--- Diff: integration/spark2/src/test/scala/org/apache/spark/carbondata/restructure/AlterTableValidationTestCase.scala ---
@@ -544,6 +546,13 @@ class AlterTableValidationTestCase extends Spark2QueryTest with BeforeAndAfterAl
sql("drop table if exists restructure1")
sql("drop table if exists restructure")
}
+test("test alter command for boolean data type with correct default measure value") {
+ sql("create table testalterwithboolean(id int,name string) stored by 'carbondata' ")
+ sql("insert into testalterwithboolean values(1,'anubhav') ")
+ sql(
+ "alter table testalterwithboolean add columns(booleanfield boolean) tblproperties('default.value.booleanfield'='true')")
+ checkAnswer(sql("select * from testalterwithboolean"),Seq(Row(1,"anubhav",true)))
+}
--- End diff --
done
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on the issue:
https://github.com/apache/carbondata/pull/1934
@anubhav100 ... In your PR description above no need to mention about the code details...that will be reviewed as part of your PR....kindly remove the code details from PR description and follow the PR description template
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1934
retest sdv please
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1934
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3533/
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1934
retest sdv please
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1934
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3544/
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1934
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3541/
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1934
@manishgupta88 i have done changes please check
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1934
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3375/
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1934
retest sdv please
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1934
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3742/
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1934
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2297/
---
[GitHub] carbondata pull request #1934: [CARBONDATA-2133] Fixed Exception displays af...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1934
---
[GitHub] carbondata pull request #1934: [CARBONDATA-2133] Fixed Exception displays af...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1934#discussion_r168083518
--- Diff: integration/spark2/src/test/scala/org/apache/spark/carbondata/restructure/AlterTableValidationTestCase.scala ---
@@ -544,6 +546,13 @@ class AlterTableValidationTestCase extends Spark2QueryTest with BeforeAndAfterAl
sql("drop table if exists restructure1")
sql("drop table if exists restructure")
}
+test("test alter command for boolean data type with correct default measure value") {
+ sql("create table testalterwithboolean(id int,name string) stored by 'carbondata' ")
+ sql("insert into testalterwithboolean values(1,'anubhav') ")
+ sql(
+ "alter table testalterwithboolean add columns(booleanfield boolean) tblproperties('default.value.booleanfield'='true')")
+ checkAnswer(sql("select * from testalterwithboolean"),Seq(Row(1,"anubhav",true)))
+}
--- End diff --
Please add one more test case for boolean type where the default value is not provided
---
[GitHub] carbondata pull request #1934: [CARBONDATA-2133] Fixed Exception displays af...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1934#discussion_r168083426
--- Diff: core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java ---
@@ -720,21 +721,22 @@ private static String parseStringToBigDecimal(String value, ColumnSchema columnS
return null;
}
+ public static DataTypeConverter getDataTypeConverter() {
+ if (converter == null) {
+ converter = new DataTypeConverterImpl();
+ }
+ return converter;
+ }
+
--- End diff --
This change does not belong to this PR...the code is already present in the same class...kindly remove all the changes that does not belong to this PR
---
[GitHub] carbondata issue #1934: [CARBONDATA-2133] Fixed Exception displays after per...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on the issue:
https://github.com/apache/carbondata/pull/1934
LGTM
---