You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by GitBox <gi...@apache.org> on 2021/06/04 14:46:26 UTC
[GitHub] [carbondata] ShreelekhyaG opened a new pull request #4147: [WIP] Fix issue when refresh main table with MV
ShreelekhyaG opened a new pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147
### Why is this PR needed?
When trying to register a table of old store which has MV, it fails parser error(syntax issue while creating table). It is trying to create table with `relatedmvtablesmap` property which is not valid.
### What changes were proposed in this PR?
- Removed `relatedmvtablesmap` from table properties in `RefreshCarbonTableCommand`
- After Main table has registered, to register MV made changes to get the schema from the system folder and register.
### Does this PR introduce any user interface change?
- No
- Yes. (please explain the change and update document)
### Is any new testcase added?
- Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856514647
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856560360
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5519/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856867436
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5526/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856654750
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5522/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-858387425
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3792/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856665109
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#discussion_r647299409
##########
File path: integration/spark/src/test/scala/org/apache/spark/carbondata/register/TestRegisterCarbonTable.scala
##########
@@ -231,6 +232,45 @@ class TestRegisterCarbonTable extends QueryTest with BeforeAndAfterEach {
}
}
+ test("test register table with mv") {
+ sql(s"create database carbon location '$dbLocation'")
+ sql("use carbon")
+ sql(
+ """create table carbon.carbontable (
+ |c1 string,c2 int,c3 string,c5 string) STORED AS carbondata""".stripMargin)
+ sql("insert into carbontable select 'a',1,'aa','aaa'")
+ sql("create materialized view mv1 as select avg(c2),c3 from carbontable group by c3")
+ backUpData(dbLocation, Some("carbon"), "carbontable")
+ backUpData(dbLocation, Some("carbon"), "mv1")
+ val source = dbLocation + CarbonCommonConstants.FILE_SEPARATOR + "_system" +
+ CarbonCommonConstants.FILE_SEPARATOR + "mv_schema.mv1"
+ val destination = dbLocation + "_back" + CarbonCommonConstants.FILE_SEPARATOR + "mv_schema.mv1"
+ backUpMvSchema(source, destination)
+ sql("drop table carbontable")
+ if (!CarbonEnv.getInstance(sqlContext.sparkSession).carbonMetaStore.isReadFromHiveMetaStore) {
+ restoreData(dbLocation, "carbontable")
+ restoreData(dbLocation, "mv1")
+ backUpMvSchema(destination, source)
+ sql("refresh table carbontable")
+ sql("refresh table mv1")
+ sql("refresh materialized view mv1")
+ checkAnswer(sql("select count(*) from carbontable"), Row(1))
+ checkAnswer(sql("select c1 from carbontable"), Seq(Row("a")))
+ sql("show materialized views on table carbontable").show(false)
Review comment:
please remove this show
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [WIP] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-854858593
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3763/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856745833
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856489014
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-858311083
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [WIP] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-854902879
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5506/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856656830
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3779/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#discussion_r647969173
##########
File path: integration/spark/src/test/scala/org/apache/spark/carbondata/register/TestRegisterCarbonTable.scala
##########
@@ -231,6 +232,45 @@ class TestRegisterCarbonTable extends QueryTest with BeforeAndAfterEach {
}
}
+ test("test register table with mv") {
+ sql(s"create database carbon location '$dbLocation'")
+ sql("use carbon")
+ sql(
+ """create table carbon.carbontable (
+ |c1 string,c2 int,c3 string,c5 string) STORED AS carbondata""".stripMargin)
+ sql("insert into carbontable select 'a',1,'aa','aaa'")
+ sql("create materialized view mv1 as select avg(c2),c3 from carbontable group by c3")
+ backUpData(dbLocation, Some("carbon"), "carbontable")
+ backUpData(dbLocation, Some("carbon"), "mv1")
+ val source = dbLocation + CarbonCommonConstants.FILE_SEPARATOR + "_system" +
+ CarbonCommonConstants.FILE_SEPARATOR + "mv_schema.mv1"
+ val destination = dbLocation + "_back" + CarbonCommonConstants.FILE_SEPARATOR + "mv_schema.mv1"
+ backUpMvSchema(source, destination)
+ sql("drop table carbontable")
+ if (!CarbonEnv.getInstance(sqlContext.sparkSession).carbonMetaStore.isReadFromHiveMetaStore) {
+ restoreData(dbLocation, "carbontable")
+ restoreData(dbLocation, "mv1")
+ backUpMvSchema(destination, source)
+ sql("refresh table carbontable")
+ sql("refresh table mv1")
+ sql("refresh materialized view mv1")
+ checkAnswer(sql("select count(*) from carbontable"), Row(1))
+ checkAnswer(sql("select c1 from carbontable"), Seq(Row("a")))
+ sql("show materialized views on table carbontable").show(false)
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856726384
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3781/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856563176
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3776/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-856864464
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3783/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4147: [CARBONDATA-4202] Fix issue when refresh main table with MV
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4147:
URL: https://github.com/apache/carbondata/pull/4147#issuecomment-858404376
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5535/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org