You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by GitBox <gi...@apache.org> on 2021/08/19 15:10:43 UTC
[GitHub] [carbondata] ShreelekhyaG opened a new pull request #4208: [WIP] Fix create partition table error with spark 3.1
ShreelekhyaG opened a new pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208
### Why is this PR needed?
With spark 3.1, we can create a partition table by giving partition columns from schema.
Like below example:
`create table partitionTable(c1 int, c2 int, v1 string, v2 string) stored as carbondata partitioned by (v2,c2)`
In cluster/ with carbon session, when we create partition table with above syntax it is creating normal table with no partitions.
### What changes were proposed in this PR?
`partitionByStructFields` is empty when we directly give partition column names. So it was not creating a partition table. Made changes to identify the partition column names and get the struct field and datatype info from table columns.
### Does this PR introduce any user interface change?
- No
### Is any new testcase added?
- Yes, tested in cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-905196832
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5886/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] MarvinLitt edited a comment on pull request #4208: [WIP] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
MarvinLitt edited a comment on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-902354173
how does the CI pass under spark3.1?
the partition table test case should work to take care of this issue, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on a change in pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#discussion_r695604258
##########
File path: integration/spark/src/main/spark3.1/org/apache/spark/sql/parser/CarbonSparkSqlParser.scala
##########
@@ -130,8 +131,24 @@ class CarbonHelperSqlAstBuilder(conf: SQLConf,
val tableProperties = convertPropertiesToLowercase(properties)
// validate partition clause
- val partitionByStructFields = Option(partitionColumns).toSeq
- .flatMap(x => visitPartitionFieldList(x)._2)
+ var (partitionTransformList,
+ partitionByStructFields) = visitPartitionFieldList(partitionColumns)
+ // If the struct fields is empty but partition columns are given,
+ // then search in table columns to extract the struct fields.
+ if (partitionByStructFields.isEmpty && !partitionTransformList.isEmpty) {
Review comment:
change !Seq().empty to Seq().nonEmpty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4208: [WIP] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-903235354
> how does the CI pass under spark3.1?
> the partition table test case should work to take care of this issue, right?
Prior to this, there is no such test case in CI with syntax like below, where we can create a partition table by giving partition column names from schema.
`create table partitionTable(c1 int, c2 int, v1 string, v2 string) stored as carbondata partitioned by (v2,c2)`
With 2.3 and 2.4 versions, the above syntax will fail while parsing and only the syntax with partition column names and datatype is valid like below.
`create table partitionTable(c1 int, v1 string) stored as carbondata partitioned by (v2 string,c2 int)`
spark 3.1 supports creation of partition table with both of the above syntax types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
kunal642 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-908896905
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-908967221
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4158/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-905191500
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/288/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-908947456
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5903/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
kunal642 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-908896905
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on a change in pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#discussion_r695602945
##########
File path: integration/spark/src/main/spark3.1/org/apache/spark/sql/parser/CarbonSparkSqlParser.scala
##########
@@ -130,8 +131,24 @@ class CarbonHelperSqlAstBuilder(conf: SQLConf,
val tableProperties = convertPropertiesToLowercase(properties)
// validate partition clause
- val partitionByStructFields = Option(partitionColumns).toSeq
- .flatMap(x => visitPartitionFieldList(x)._2)
+ var (partitionTransformList,
+ partitionByStructFields) = visitPartitionFieldList(partitionColumns)
+ // If the struct fields is empty but partition columns are given,
+ // then search in table columns to extract the struct fields.
+ if (partitionByStructFields.isEmpty && !partitionTransformList.isEmpty) {
+ val partitionNames = partitionTransformList
+ .flatMap(_.references().flatMap(_.fieldNames()))
+ partitionNames.foreach(partName => {
+ val ans = cols.find(x => x.name.equals(partName))
Review comment:
change var name to structField
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [WIP] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-902111116
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4132/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-905266098
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/289/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] MarvinLitt commented on pull request #4208: [WIP] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
MarvinLitt commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-902354173
how does the CI pass under spark3.1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-905545236
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4149/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-908954820
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/303/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-905531779
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/294/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [WIP] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-902008469
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/277/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-908947456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-905199284
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4143/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
kunal642 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-906147999
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [WIP] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-902103171
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5874/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-904891413
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5884/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-905200833
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-905274797
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5887/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-904871943
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/286/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-904895355
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4141/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-905276994
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4144/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#issuecomment-905539537
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5893/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#discussion_r695729008
##########
File path: integration/spark/src/main/spark3.1/org/apache/spark/sql/parser/CarbonSparkSqlParser.scala
##########
@@ -130,8 +131,24 @@ class CarbonHelperSqlAstBuilder(conf: SQLConf,
val tableProperties = convertPropertiesToLowercase(properties)
// validate partition clause
- val partitionByStructFields = Option(partitionColumns).toSeq
- .flatMap(x => visitPartitionFieldList(x)._2)
+ var (partitionTransformList,
+ partitionByStructFields) = visitPartitionFieldList(partitionColumns)
+ // If the struct fields is empty but partition columns are given,
+ // then search in table columns to extract the struct fields.
+ if (partitionByStructFields.isEmpty && !partitionTransformList.isEmpty) {
+ val partitionNames = partitionTransformList
+ .flatMap(_.references().flatMap(_.fieldNames()))
+ partitionNames.foreach(partName => {
+ val ans = cols.find(x => x.name.equals(partName))
Review comment:
Done
##########
File path: integration/spark/src/main/spark3.1/org/apache/spark/sql/parser/CarbonSparkSqlParser.scala
##########
@@ -130,8 +131,24 @@ class CarbonHelperSqlAstBuilder(conf: SQLConf,
val tableProperties = convertPropertiesToLowercase(properties)
// validate partition clause
- val partitionByStructFields = Option(partitionColumns).toSeq
- .flatMap(x => visitPartitionFieldList(x)._2)
+ var (partitionTransformList,
+ partitionByStructFields) = visitPartitionFieldList(partitionColumns)
+ // If the struct fields is empty but partition columns are given,
+ // then search in table columns to extract the struct fields.
+ if (partitionByStructFields.isEmpty && !partitionTransformList.isEmpty) {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4208: [CARBONDATA-4274] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#discussion_r695412943
##########
File path: integration/spark/src/main/spark3.1/org/apache/spark/sql/parser/CarbonSparkSqlParser.scala
##########
@@ -129,9 +129,32 @@ class CarbonHelperSqlAstBuilder(conf: SQLConf,
val properties: Map[String, String] = getPropertyKeyValues(tablePropertyList)
val tableProperties = convertPropertiesToLowercase(properties)
+ var partitionTransformList: Seq[Transform] = Seq()
+ var partitionByStructFields: Seq[StructField] = Seq()
// validate partition clause
- val partitionByStructFields = Option(partitionColumns).toSeq
- .flatMap(x => visitPartitionFieldList(x)._2)
+ var partitionTransformList: Seq[Transform] = Seq()
+ var partitionByStructFields: Seq[StructField] = Seq()
+ Option(partitionColumns).toSeq.foreach(x => {
+ val (t, s) = visitPartitionFieldList(x)
+ partitionTransformList = partitionTransformList ++ t
+ partitionByStructFields = partitionByStructFields ++ s
+ })
Review comment:
Have made the change to directly call the `visitPartitionFieldList `function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] MarvinLitt commented on a change in pull request #4208: [WIP] Fix create partition table error with spark 3.1
Posted by GitBox <gi...@apache.org>.
MarvinLitt commented on a change in pull request #4208:
URL: https://github.com/apache/carbondata/pull/4208#discussion_r693614269
##########
File path: integration/spark/src/main/spark3.1/org/apache/spark/sql/parser/CarbonSparkSqlParser.scala
##########
@@ -129,9 +129,32 @@ class CarbonHelperSqlAstBuilder(conf: SQLConf,
val properties: Map[String, String] = getPropertyKeyValues(tablePropertyList)
val tableProperties = convertPropertiesToLowercase(properties)
+ var partitionTransformList: Seq[Transform] = Seq()
+ var partitionByStructFields: Seq[StructField] = Seq()
// validate partition clause
- val partitionByStructFields = Option(partitionColumns).toSeq
- .flatMap(x => visitPartitionFieldList(x)._2)
+ var partitionTransformList: Seq[Transform] = Seq()
+ var partitionByStructFields: Seq[StructField] = Seq()
+ Option(partitionColumns).toSeq.foreach(x => {
+ val (t, s) = visitPartitionFieldList(x)
+ partitionTransformList = partitionTransformList ++ t
+ partitionByStructFields = partitionByStructFields ++ s
+ })
Review comment:
Option(partitionColumns).toSeq.foreach(x => {
val (t, s) = visitPartitionFieldList(x)
partitionTransformList = partitionTransformList ++ t
partitionByStructFields = partitionByStructFields ++ s
})
does this no need to use foreach, the input of visitPartitionFieldList is PartitionFieldListContext type and after func visitPartitionFieldList the result is two list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org