You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by GitBox <gi...@apache.org> on 2021/09/22 17:44:47 UTC
[GitHub] [carbondata] Indhumathi27 opened a new pull request #4221: [WIP] Fix partition table issue with location
Indhumathi27 opened a new pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221
### Why is this PR needed?
### What changes were proposed in this PR?
### Does this PR introduce any user interface change?
- No
- Yes. (please explain the change and update document)
### Is any new testcase added?
- No
- Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-930155335
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4216/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
kunal642 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-935911867
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-936187583
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6009/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [WIP] Fix partition table issue with location
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-925569089
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5946/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on a change in pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#discussion_r722963262
##########
File path: integration/spark/src/main/common2.3and2.4/org/apache/spark/sql/SparkVersionAdapter.scala
##########
@@ -430,6 +430,22 @@ trait SparkVersionAdapter {
def evaluateWithPredicate(exp: Expression, schema: Seq[Attribute], row: InternalRow): Any = {
InterpretedPredicate.create(exp, schema).expression.eval(row)
}
+
+ def getUpdatedPlan(plan: LogicalPlan, sqlText: String): LogicalPlan = {
+ plan match {
+ case create@CreateTable(tableDesc, mode, query) =>
+ if ( tableDesc.storage.locationUri.isDefined &&
+ !sqlText.toUpperCase.startsWith("CREATE EXTERNAL TABLE ")) {
+ // add a property to differentiate if create table statement has external keyword or not
+ val newProperties = tableDesc.properties. +("hasexternalkeyword" -> "false")
+ val updatedTableDesc = tableDesc.copy(properties = newProperties)
+ CreateTable(updatedTableDesc, mode, query)
Review comment:
can we make the table type as external , instead of putting in properties.
This was no need to check for this keyword and the existing flows can handle the transactional table creation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-935953510
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4261/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-937434569
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
kunal642 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-937694366
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [WIP] Fix partition table issue with location
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-925281438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-937434569
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-937483350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [WIP] Fix partition table issue with location
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-925570088
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4200/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#discussion_r723056567
##########
File path: integration/spark/src/main/common2.3and2.4/org/apache/spark/sql/SparkVersionAdapter.scala
##########
@@ -430,6 +430,22 @@ trait SparkVersionAdapter {
def evaluateWithPredicate(exp: Expression, schema: Seq[Attribute], row: InternalRow): Any = {
InterpretedPredicate.create(exp, schema).expression.eval(row)
}
+
+ def getUpdatedPlan(plan: LogicalPlan, sqlText: String): LogicalPlan = {
+ plan match {
+ case create@CreateTable(tableDesc, mode, query) =>
+ if ( tableDesc.storage.locationUri.isDefined &&
+ !sqlText.toUpperCase.startsWith("CREATE EXTERNAL TABLE ")) {
+ // add a property to differentiate if create table statement has external keyword or not
+ val newProperties = tableDesc.properties. +("hasexternalkeyword" -> "false")
+ val updatedTableDesc = tableDesc.copy(properties = newProperties)
+ CreateTable(updatedTableDesc, mode, query)
Review comment:
Catalog table is available only in Spark versions 2.3 and 2.4. In 3.1 version, spark will create CatalogTable after parsing. So, I think, better we can keep same behavior(current code changes) for all versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-937497628
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/414/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-937483350
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6014/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
kunal642 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-937694366
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-936020600
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4263/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-930143175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on a change in pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#discussion_r723098682
##########
File path: integration/spark/src/main/common2.3and2.4/org/apache/spark/sql/SparkVersionAdapter.scala
##########
@@ -430,6 +430,22 @@ trait SparkVersionAdapter {
def evaluateWithPredicate(exp: Expression, schema: Seq[Attribute], row: InternalRow): Any = {
InterpretedPredicate.create(exp, schema).expression.eval(row)
}
+
+ def getUpdatedPlan(plan: LogicalPlan, sqlText: String): LogicalPlan = {
+ plan match {
+ case create@CreateTable(tableDesc, mode, query) =>
+ if ( tableDesc.storage.locationUri.isDefined &&
+ !sqlText.toUpperCase.startsWith("CREATE EXTERNAL TABLE ")) {
+ // add a property to differentiate if create table statement has external keyword or not
+ val newProperties = tableDesc.properties. +("hasexternalkeyword" -> "false")
+ val updatedTableDesc = tableDesc.copy(properties = newProperties)
+ CreateTable(updatedTableDesc, mode, query)
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [WIP] Fix partition table issue with location
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-925191361
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4199/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-936073265
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/408/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-935920420
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/406/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-935977168
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-937508212
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4268/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-930143175
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/361/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [CARBONDATA-4293] Make Table created without external keyword as Transactional table
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-930157601
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5961/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4221: [WIP] Fix partition table issue with location
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4221:
URL: https://github.com/apache/carbondata/pull/4221#issuecomment-925565739
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/347/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org