You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by ravipesala <gi...@git.apache.org> on 2018/01/03 19:12:06 UTC
[GitHub] carbondata pull request #1759: [CARBONDATA-1980][PARTITION]Fix adding partit...
GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/1759
[CARBONDATA-1980][PARTITION]Fix adding partition information while doing refresh(restore) table.And fix the case sensitivity issue of partition column
Problem
While restoring the table the partitions are not added to the hive. And also if any uppercase letters are present in partition column then filter queries on that column gives error.
Solution
Read all partitionmap files of all segments to add back the partition information to the hive metatstore.
And always update the partition filter columns to lower case before querying partition information.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [X] Any interfaces changed? NO
- [X] Any backward compatibility impacted? NO
- [X] Document update required? NO
- [X] Testing done
Tests added
- [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/incubator-carbondata restore-partition
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1759.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1759
----
commit 20e411487a11c2219e08e966ff83c6cc0e718678
Author: ravipesala <ra...@...>
Date: 2018-01-03T19:04:33Z
Fix adding partition information while doing refresh(restore) table. And fix the case sensitivity issue of partition column
----
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1759
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2741/
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1759
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2556/
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1759
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2702/
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1759
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2740/
---
[GitHub] carbondata pull request #1759: [CARBONDATA-1980][PARTITION]Fix adding partit...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala closed the pull request at:
https://github.com/apache/carbondata/pull/1759
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1759
Already merged
---
[GitHub] carbondata pull request #1759: [CARBONDATA-1980][PARTITION]Fix adding partit...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1759#discussion_r159669950
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/RefreshCarbonTableCommand.scala ---
@@ -91,6 +95,11 @@ case class RefreshCarbonTableCommand(
registerAggregates(databaseName, dataMapSchemaList)(sparkSession)
}
registerTableWithHive(databaseName, tableName, tableInfo)(sparkSession)
+ // Register partitions to hive metastore in case of hive partitioning carbon table
+ if (tableInfo.getFactTable.getPartitionInfo != null &&
+ tableInfo.getFactTable.getPartitionInfo.getPartitionType == PartitionType.NATIVE_HIVE) {
+ registerAllPartitionsToHive(tablePath, databaseName, tableName, sparkSession)
--- End diff --
ok
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1759
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1300/
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1759
LGTM after fixing comment and rebasing
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1759
retest sdv please
---
[GitHub] carbondata pull request #1759: [CARBONDATA-1980][PARTITION]Fix adding partit...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1759#discussion_r159589011
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/RefreshCarbonTableCommand.scala ---
@@ -91,6 +95,11 @@ case class RefreshCarbonTableCommand(
registerAggregates(databaseName, dataMapSchemaList)(sparkSession)
}
registerTableWithHive(databaseName, tableName, tableInfo)(sparkSession)
+ // Register partitions to hive metastore in case of hive partitioning carbon table
+ if (tableInfo.getFactTable.getPartitionInfo != null &&
+ tableInfo.getFactTable.getPartitionInfo.getPartitionType == PartitionType.NATIVE_HIVE) {
+ registerAllPartitionsToHive(tablePath, databaseName, tableName, sparkSession)
--- End diff --
I think passing AbsolutionTableIdentifier is better, it has all information. It is created in line 65
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1759
LGTM
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1759
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2540/
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1759
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2576/
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1759
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1340/
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1759
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1332/
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1759
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1311/
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1759
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2530/
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1759
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2694/
---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1759
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2730/
---