You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by anubhav100 <gi...@git.apache.org> on 2018/01/19 16:03:08 UTC
[GitHub] carbondata pull request #1839: [CARBONDATA-2016] Exception displays while ex...
GitHub user anubhav100 opened a pull request:
https://github.com/apache/carbondata/pull/1839
[CARBONDATA-2016] Exception displays while executing compaction with alter query
**Root Cause**
When we apply the alter table command to add column with default value it is storing it as long object,it is wrongly written in restructure util we should get the value as the same type
as that of datatype of columnschema in restructure util earlier in master branch in restuctureutil class if our data type is long or short or int we are always returning back a long object which is wong due to same reason compaction was failing
if it was applied after alter table add columns command with default value because in sortdatarows there was mismatch between data type and its corresponding value
**Testing:**
1.mvn clean install is passing
2.added new test case for same
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/anubhav100/incubator-carbondata bugfix/CARBONDATA-2016
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1839.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1839
----
commit eee0a2325b410c9d1f9e18b40002446b067ed581
Author: anubhav100 <an...@...>
Date: 2018-01-19T16:01:14Z
When we apply the alter table command to add column with default value it is storing it as long object,it is wrongly written in restructure util we should get the value as the same type
----
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1839
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1775/
---
[GitHub] carbondata pull request #1839: [CARBONDATA-2016] Exception displays while ex...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1839#discussion_r164339675
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/HorizontalCompactionTestCase.scala ---
@@ -350,8 +350,41 @@ class HorizontalCompactionTestCase extends QueryTest with BeforeAndAfterAll {
)
sql("""drop table dest2""")
}
+ test("test the compaction after alter command") // As per bug Carbondata-2016
+ {
+ try {
+ sql(
+ "CREATE TABLE CUSTOMER1 ( C_CUSTKEY INT , C_NAME STRING , C_ADDRESS STRING , C_NATIONKEY INT , C_PHONE STRING , C_ACCTBAL DECIMAL(15,2) , C_MKTSEGMENT STRING , C_COMMENT STRING) stored by 'carbondata'")
+ sql(
+ "insert into customer1 values(1,'vandana','noida',1,'123456789',45987.78,'hello','comment')")
+ sql(
+ "insert into customer1 values(2,'vandana','noida',2,'123456789',487.78,'hello','comment')")
+
+ sql(
+ " insert into customer1 values(3,'geetika','delhi',3,'123456789',487897.78,'hello','comment')")
+
+ sql(
+ "insert into customer1 values(4,'sangeeta','delhi',3,'123456789',48789.78,'hello','comment')")
+
+ sql(
+ "alter table customer1 add columns (shortfield short) TBLPROPERTIES ('DEFAULT.VALUE.shortfield'='10')")
+
+ sql(
+ "alter table customer1 add columns (intfield int) TBLPROPERTIES ('DEFAULT.VALUE.intfield'='10')")
+
+ sql(
+ "alter table customer1 add columns (longfield bigint) TBLPROPERTIES ('DEFAULT.VALUE.longfield'='10')")
+
+ sql("alter table customer1 compact 'minor' ").show()
+ assert(true)
--- End diff --
1. Remove try catch block..its not required..in case of any exception anyways the test case will fail.
2. In the add column DDL's. give the default value as MAX value of the corresponding datatype so that the same is validated after compaction in the select query validation.
3. After compaction match the newly added column values as select query using checkAnswer.
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1839
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1770/
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1839
@jackylk please review
---
[GitHub] carbondata pull request #1839: [CARBONDATA-2016] Exception displays while ex...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1839#discussion_r164339708
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/HorizontalCompactionTestCase.scala ---
@@ -360,6 +393,8 @@ class HorizontalCompactionTestCase extends QueryTest with BeforeAndAfterAll {
CarbonProperties.getInstance()
.addProperty(CarbonCommonConstants.isHorizontalCompactionEnabled , "true")
sql("""drop table if exists t_carbn01""")
+ sql("""drop table if exists customer1""")
+
--- End diff --
Remove extra line
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1839
retest this please
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1839
retest this please
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1839
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1774/
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on the issue:
https://github.com/apache/carbondata/pull/1839
LGTM
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1839
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3213/
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1839
@jackylk i accidently deleted my old branch i have created this new branch for same issue please merge this one
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1839
@jackylk can merge this one
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on the issue:
https://github.com/apache/carbondata/pull/1839
retest this please
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1839
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3100/
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1839
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3174/
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1839
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1979/
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1839
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3005/
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1839
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3000/
---
[GitHub] carbondata pull request #1839: [CARBONDATA-2016] Exception displays while ex...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1839
---
[GitHub] carbondata pull request #1839: [CARBONDATA-2016] Exception displays while ex...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1839#discussion_r164344094
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/HorizontalCompactionTestCase.scala ---
@@ -360,6 +393,8 @@ class HorizontalCompactionTestCase extends QueryTest with BeforeAndAfterAll {
CarbonProperties.getInstance()
.addProperty(CarbonCommonConstants.isHorizontalCompactionEnabled , "true")
sql("""drop table if exists t_carbn01""")
+ sql("""drop table if exists customer1""")
+
--- End diff --
@manishgupta88 done you can review
---
[GitHub] carbondata pull request #1839: [CARBONDATA-2016] Exception displays while ex...
Posted by anubhav100 <gi...@git.apache.org>.
Github user anubhav100 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1839#discussion_r164344032
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/HorizontalCompactionTestCase.scala ---
@@ -350,8 +350,41 @@ class HorizontalCompactionTestCase extends QueryTest with BeforeAndAfterAll {
)
sql("""drop table dest2""")
}
+ test("test the compaction after alter command") // As per bug Carbondata-2016
+ {
+ try {
+ sql(
+ "CREATE TABLE CUSTOMER1 ( C_CUSTKEY INT , C_NAME STRING , C_ADDRESS STRING , C_NATIONKEY INT , C_PHONE STRING , C_ACCTBAL DECIMAL(15,2) , C_MKTSEGMENT STRING , C_COMMENT STRING) stored by 'carbondata'")
+ sql(
+ "insert into customer1 values(1,'vandana','noida',1,'123456789',45987.78,'hello','comment')")
+ sql(
+ "insert into customer1 values(2,'vandana','noida',2,'123456789',487.78,'hello','comment')")
+
+ sql(
+ " insert into customer1 values(3,'geetika','delhi',3,'123456789',487897.78,'hello','comment')")
+
+ sql(
+ "insert into customer1 values(4,'sangeeta','delhi',3,'123456789',48789.78,'hello','comment')")
+
+ sql(
+ "alter table customer1 add columns (shortfield short) TBLPROPERTIES ('DEFAULT.VALUE.shortfield'='10')")
+
+ sql(
+ "alter table customer1 add columns (intfield int) TBLPROPERTIES ('DEFAULT.VALUE.intfield'='10')")
+
+ sql(
+ "alter table customer1 add columns (longfield bigint) TBLPROPERTIES ('DEFAULT.VALUE.longfield'='10')")
+
+ sql("alter table customer1 compact 'minor' ").show()
+ assert(true)
--- End diff --
@manishgupta88 done
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1839
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1889/
---
[GitHub] carbondata issue #1839: [CARBONDATA-2016] Exception displays while executing...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1839
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3121/
---