You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by anubhav100 <gi...@git.apache.org> on 2018/01/11 10:13:54 UTC
[GitHub] carbondata pull request #1795: [CARBONDATA-2016] Exception displays while ex...
GitHub user anubhav100 opened a pull request:
https://github.com/apache/carbondata/pull/1795
[CARBONDATA-2016] Exception displays while executing compaction with alter query
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/anubhav100/incubator-carbondata CARBONDATA-2016
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1795.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1795
----
commit 94a1df25bf2165ff6fcd0ffa66051664f8115998
Author: anubhav100 <an...@...>
Date: 2018-01-11T10:08:49Z
Exception displays while executing compaction with alter query
commit 476d659b6e4cf0b66f87194778854ef75d593eea
Author: anubhav100 <an...@...>
Date: 2018-01-11T10:11:09Z
Revert "Exception displays while executing compaction with alter query"
This reverts commit 94a1df25bf2165ff6fcd0ffa66051664f8115998.
commit 43d6368c1567c537ea6b0f4f70a02a9d5d51c079
Author: anubhav100 <an...@...>
Date: 2018-01-11T10:12:15Z
Exception displays while executing compaction with alter query
----
---
[GitHub] carbondata pull request #1795: [CARBONDATA-2016] Exception displays while ex...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1795#discussion_r161792809
--- Diff: processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SortDataRows.java ---
@@ -270,7 +270,7 @@ private void writeDataToFile(Object[][] recordHolderList, int entryCountLocal, F
} else if (dataType == DataTypes.SHORT) {
stream.writeShort((Short) value);
} else if (dataType == DataTypes.INT) {
- stream.writeInt((Integer) value);
+ stream.writeInt(Integer.parseInt(value.toString()));
--- End diff --
But what is the root cause of this bug? I think your change (convert to string and then int) is like hiding this bug.
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1795
retest this please
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1795
@jackylk
**here is the root reason**
**when we apply the alter table command with default value i;e
"alter table customer1 add columns (intfield int) TBLPROPERTIES ('DEFAULT.VALUE.intfield'='10')")
that value is always return as the long object in restructure util thats why compaction failed because the column value for newly added column is long but the column type is int**
it is wrongly written in restructure util this code is wrong
public static Object getMeasureDefaultValue(ColumnSchema columnSchema, byte[] defaultValue) {
Object measureDefaultValue = null;
if (!isDefaultValueNull(defaultValue)) {
String value = null;
DataType dataType = columnSchema.getDataType();
if (dataType == DataTypes.SHORT || dataType == DataTypes.INT || dataType == DataTypes.LONG) {
value = new String(defaultValue, Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
measureDefaultValue = Long.parseLong(value);
} else if (DataTypes.isDecimal(dataType)) {
BigDecimal decimal = DataTypeUtil.byteToBigDecimal(defaultValue);
if (columnSchema.getScale() > decimal.scale()) {
decimal = decimal.setScale(columnSchema.getScale(), RoundingMode.HALF_UP);
}
measureDefaultValue = decimal;
} else {
value = new String(defaultValue, Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
Double parsedValue = Double.valueOf(value);
if (!Double.isInfinite(parsedValue) && !Double.isNaN(parsedValue)) {
measureDefaultValue = parsedValue;
}
}
}
return measureDefaultValue;
}
this code is always returning a long value objetc that is wrong
i corrected it as
*/
public static Object getMeasureDefaultValue(ColumnSchema columnSchema, byte[] defaultValue) {
Object measureDefaultValue = null;
if (!isDefaultValueNull(defaultValue)) {
String value = null;
DataType dataType = columnSchema.getDataType();
if (dataType == DataTypes.SHORT) {
value = new String(defaultValue, Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
measureDefaultValue = new Short(value);
}
else if (dataType == DataTypes.LONG) {
value = new String(defaultValue, Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
measureDefaultValue = Long.parseLong(value);
}
else if (dataType == DataTypes.INT) {
value = new String(defaultValue, Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
measureDefaultValue = Integer.parseInt(value);
}
else if (DataTypes.isDecimal(dataType)) {
BigDecimal decimal = DataTypeUtil.byteToBigDecimal(defaultValue);
if (columnSchema.getScale() > decimal.scale()) {
decimal = decimal.setScale(columnSchema.getScale(), RoundingMode.HALF_UP);
}
measureDefaultValue = decimal;
} else {
value = new String(defaultValue, Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
Double parsedValue = Double.valueOf(value);
if (!Double.isInfinite(parsedValue) && !Double.isNaN(parsedValue)) {
measureDefaultValue = parsedValue;
}
}
}
return measureDefaultValue;
}
please review i corrected it
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1473/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1676/
---
[GitHub] carbondata pull request #1795: [CARBONDATA-2016] Exception displays while ex...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1795#discussion_r161663046
--- Diff: processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SortDataRows.java ---
@@ -270,7 +270,7 @@ private void writeDataToFile(Object[][] recordHolderList, int entryCountLocal, F
} else if (dataType == DataTypes.SHORT) {
stream.writeShort((Short) value);
} else if (dataType == DataTypes.INT) {
- stream.writeInt((Integer) value);
+ stream.writeInt(Integer.parseInt(value.toString()));
--- End diff --
here is exception trace 18/01/10 16:17:12 ERROR CompactionResultSortProcessor: [Executor task launch worker-36][partitionID:customer1;queryID:15798380253871] Compaction failed: java.lang.Long cannot be cast to java.lang.Integer
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Integer
at org.apache.carbondata.processing.sort.sortdata.SortDataRows.writeDataToFile(SortDataRows.java:273)
at org.apache.carbondata.processing.sort.sortdata.SortDataRows.startSorting(SortDataRows.java:214)
at org.apache.carbondata.processing.merger.CompactionResultSortProcessor.processResult(CompactionResultSortProcessor.java:226)
at org.apache.carbondata.processing.merger.CompactionResultSortProcessor.execute(CompactionResultSortProcessor.java:159)
at org.apache.carbondata.spark.rdd.CarbonMergerRDD$$anon$1.<init>(CarbonMergerRDD.scala:234)
at org.apache.carbondata.spark.rdd.CarbonMergerRDD.internalCompute(CarbonMergerRDD.scala:81)
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2703/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1795
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2838/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2712/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1795
@jackylk i accidently deleted the commits of this branch when squashing the commit i raised the new pull request with same changes please review #1839
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2914/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1795
retest this please
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2711/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2921/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1795
retest this please
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2709/
---
[GitHub] carbondata pull request #1795: [CARBONDATA-2016] Exception displays while ex...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1795#discussion_r161794151
--- Diff: processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SortDataRows.java ---
@@ -270,7 +270,7 @@ private void writeDataToFile(Object[][] recordHolderList, int entryCountLocal, F
} else if (dataType == DataTypes.SHORT) {
stream.writeShort((Short) value);
} else if (dataType == DataTypes.INT) {
- stream.writeInt((Integer) value);
+ stream.writeInt(Integer.parseInt(value.toString()));
--- End diff --
@jackylyk ok i get your point you mean we should handle it at the base root level ok I am looking into it
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1795
@jackylk please review
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2706/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1795
retest this please
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1681/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1795
please squash and push again
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1688/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2896/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2904/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1479/
---
[GitHub] carbondata pull request #1795: [CARBONDATA-2016] Exception displays while ex...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 closed the pull request at:
https://github.com/apache/carbondata/pull/1795
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1795
retest this please
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1662/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1795
retest this please
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2909/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1795
retest sdv please
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1795
retest this please
---
[GitHub] carbondata pull request #1795: [CARBONDATA-2016] Exception displays while ex...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1795#discussion_r161242586
--- Diff: processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SortDataRows.java ---
@@ -270,7 +270,7 @@ private void writeDataToFile(Object[][] recordHolderList, int entryCountLocal, F
} else if (dataType == DataTypes.SHORT) {
stream.writeShort((Short) value);
} else if (dataType == DataTypes.INT) {
- stream.writeInt((Integer) value);
+ stream.writeInt(Integer.parseInt(value.toString()));
--- End diff --
Can you post the error message and describe what is the root cause of this bug?
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1477/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1478/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1795
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3008/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1795
LGTM
---
[GitHub] carbondata pull request #1795: [CARBONDATA-2016] Exception displays while ex...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1795#discussion_r161446115
--- Diff: processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SortDataRows.java ---
@@ -270,7 +270,7 @@ private void writeDataToFile(Object[][] recordHolderList, int entryCountLocal, F
} else if (dataType == DataTypes.SHORT) {
stream.writeShort((Short) value);
} else if (dataType == DataTypes.INT) {
- stream.writeInt((Integer) value);
+ stream.writeInt(Integer.parseInt(value.toString()));
--- End diff --
https://issues.apache.org/jira/browse/CARBONDATA-2016
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1795
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2970/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1470/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1795
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2963/
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1795
retest this please
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1795
retest this please
---
[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1795
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1672/
---