You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by ravipesala <gi...@git.apache.org> on 2018/01/18 14:09:12 UTC
[GitHub] carbondata pull request #1833: [CARBONDATA-2036] Fix the insert static parti...
GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/1833
[CARBONDATA-2036] Fix the insert static partition with integer values prefix with 0 not working
When trying to insert overwrite on the static partition with 0 at first on int column has an issue.
Example :
create table test(d1 string) partition by (c1 int, c2 int, c3 int)
And use insert overwrite table partition(01, 02, 03) select "s1"
The above case has a problem as 01 is not converting to an actual integer to partition map file.
Solution :
Convert the partition values to corresponding datatype value before adding to partition file.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [X] Any interfaces changed?
- [X] Any backward compatibility impacted?
- [X] Document update required?
- [X] Testing done
Tests added
- [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/incubator-carbondata static-partition-int-issue
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1833.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1833
----
commit e9a925e36f8e44f6066853542cef3487dffec63c
Author: ravipesala <ra...@...>
Date: 2018-01-18T13:21:50Z
Fix the insert static partition with integer values prefix with 0 not working
----
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1833
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2989/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1833
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3010/
---
[GitHub] carbondata pull request #1833: [CARBONDATA-2036] Fix the insert static parti...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1833#discussion_r162600231
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/standardpartition/StandardPartitionTableOverwriteTestCase.scala ---
@@ -155,9 +155,28 @@ class StandardPartitionTableOverwriteTestCase extends QueryTest with BeforeAndAf
checkAnswer(sql("select count(*) from weather6"), Seq(Row(2)))
}
+ test("Test overwrite static partition with wrong int value") {
+ sql(
+ """
+ | CREATE TABLE weather7 (type String)
+ | PARTITIONED BY (year int, month int, day int)
+ | STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+
+ sql("insert into weather7 partition(year=2014, month=05, day=25) select 'rainy'")
+ sql("insert into weather7 partition(year=2014, month=04, day=23) select 'cloudy'")
+ sql("insert overwrite table weather7 partition(year=2014, month=05, day=25) select 'sunny'")
+ checkExistence(sql("select * from weather7"), true, "sunny")
+ checkAnswer(sql("select count(*) from weather7"), Seq(Row(2)))
+ sql("insert into weather7 partition(year=2014, month, day) select 'rainy1',06,25")
+ sql("insert into weather7 partition(year=2014, month=01, day) select 'rainy2',27")
+ sql("insert into weather7 partition(year=2014, month=01, day=02) select 'rainy3'")
+ checkAnswer(sql("select count(*) from weather7 where month=1"), Seq(Row(2)))
+ }
+
override def afterAll = {
- dropTable
+// dropTable
--- End diff --
Please uncomment this line
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1833
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3001/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1754/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on the issue:
https://github.com/apache/carbondata/pull/1833
retest this please
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1757/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2999/
---
[GitHub] carbondata pull request #1833: [CARBONDATA-2036] Fix the insert static parti...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1833#discussion_r162419365
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala ---
@@ -243,7 +243,7 @@ case class CarbonLoadDataCommand(
} catch {
case CausedBy(ex: NoRetryException) =>
// update the load entry in table status file for changing the status to marked for delete
- if (isUpdateTableStatusRequired) {
+ if (isUpdateTableStatusRequired && !table.isHivePartitionTable) {
--- End diff --
This check is not required because isUpdateTableStatusRequired flag will be set to true only if the table is not a partition table
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2989/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1833
retest sdv please
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1759/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1765/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on the issue:
https://github.com/apache/carbondata/pull/1833
LGTM
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1833
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3004/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2995/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1769/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on the issue:
https://github.com/apache/carbondata/pull/1833
retest sdv please
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1833
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2997/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1833
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3009/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2975/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1772/
---
[GitHub] carbondata pull request #1833: [CARBONDATA-2036] Fix the insert static parti...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1833#discussion_r162545921
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala ---
@@ -243,7 +243,7 @@ case class CarbonLoadDataCommand(
} catch {
case CausedBy(ex: NoRetryException) =>
// update the load entry in table status file for changing the status to marked for delete
- if (isUpdateTableStatusRequired) {
+ if (isUpdateTableStatusRequired && !table.isHivePartitionTable) {
--- End diff --
ok
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2960/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1745/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2984/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2987/
---
[GitHub] carbondata pull request #1833: [CARBONDATA-2036] Fix the insert static parti...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1833
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1732/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1833
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3006/
---
[GitHub] carbondata issue #1833: [CARBONDATA-2036] Fix the insert static partition wi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1833
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3002/
---