You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by sounakr <gi...@git.apache.org> on 2017/04/03 09:42:39 UTC
[GitHub] incubator-carbondata pull request #723: [CARBONDATA-845] Insert Select Into ...
GitHub user sounakr opened a pull request:
https://github.com/apache/incubator-carbondata/pull/723
[CARBONDATA-845] Insert Select Into Same Table
Problem : Insert Select Implementation in current Spark-2.1 prohibits to read and write into same table.
Analysis : This restriction is imposed by spark in case InsertableRelation is extended. Current insert select implementation in spark-2.1 CarbonDatasourceHadoopRelation extends InsertableRelation and override the insert method. Spark rules in case of InsertableRelation checks if source and target tables are same, then it fails and prompts "Cannot insert overwrite into table that is also being read from."
Fix:: Spark 1.5 implementation of insert select in carbon allows reading and writing into same table, so as Hive and Parquet. Therfore as part of current fix reimplemented insert-select logic of Spark-2.1 similar to Spark-1.5.
a) CarbonDatasourceHadoopRelation doesn't extends InsertableRelation.
b) New CarbonPreInsertionCasts Rule is written for spark-2.1 (For Spark-1.5 it was already present).
d) This new Rule is registered CarbonSessionState,
c) During Insert Select processing the Logical Plan wrapped by InsertIntoCarbonTable and later execute LoadTableByInsert to process insert select.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sounakr/incubator-carbondata insert_select
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/723.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #723
----
commit beb26625dc8698574f9be4701a0b80504097f049
Author: sounakr <so...@gmail.com>
Date: 2017-04-03T07:18:20Z
Insert Select Into Same Table
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #723: [CARBONDATA-845] Insert Select Into Same Ta...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/723
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1419/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #723: [CARBONDATA-845] Insert Select Into Same Ta...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/incubator-carbondata/pull/723
Can't we have same restriction like Spark, is it really required to override this behavior?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #723: [CARBONDATA-845] Insert Select Into ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/723#discussion_r109964888
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/DDLStrategy.scala ---
@@ -54,6 +54,9 @@ class DDLStrategy(sparkSession: SparkSession) extends SparkStrategy {
identifier.table.toLowerCase)) :: Nil
case ShowLoadsCommand(databaseName, table, limit) =>
ExecutedCommandExec(ShowLoads(databaseName, table.toLowerCase, limit, plan.output)) :: Nil
+ case InsertIntoCarbonTable(relation: CarbonDatasourceHadoopRelation,
+ _, child: LogicalPlan, _, _) =>
--- End diff --
Please format it properly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #723: [CARBONDATA-845] Insert Select Into Same Ta...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/incubator-carbondata/pull/723
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #723: [CARBONDATA-845] Insert Select Into Same Ta...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/723
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1406/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #723: [CARBONDATA-845] Insert Select Into ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-carbondata/pull/723
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #723: [CARBONDATA-845] Insert Select Into Same Ta...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/723
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1453/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #723: [CARBONDATA-845] Insert Select Into ...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/723#discussion_r109574552
--- Diff: integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonPreInsertionCasts.scala ---
@@ -0,0 +1,62 @@
+/*
--- End diff --
InsertInto function is a part of IUD feature. No need to create a new file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #723: [CARBONDATA-845] Insert Select Into Same Ta...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on the issue:
https://github.com/apache/incubator-carbondata/pull/723
@ravipesala Insert Select from same table is allowed both in Hive and Parquet. Spark-5746 introduced this behavior as they are deleting the source table data first before inserting when overwrite is turned ON. But for carbon insert select Overwrite is always false and as other data sources are allowing insert select into same table it is good to have. Also the below kind of queries will help to load data.
insert into tab1 sel a *4, b*4 from tab1;
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #723: [CARBONDATA-845] Insert Select Into ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/723#discussion_r109635229
--- Diff: integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonPreInsertionCasts.scala ---
@@ -0,0 +1,62 @@
+/*
--- End diff --
Done, currently to keep 1.5 and 2.1 in sync, CarbonAnalysisRules.scala will keep the rules of IUD and Insert Select. Created CarbonAnalysisRules.scala in 2.1 where currenly on Insert-Select is there as IUD is still not ported to 2.1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #723: [CARBONDATA-845] Insert Select Into Same Ta...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/incubator-carbondata/pull/723
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---