You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by GitBox <gi...@apache.org> on 2021/06/21 13:47:04 UTC

[GitHub] [carbondata] Indhumathi27 opened a new pull request #4156: [WIP] Fix Update performance issues when auto merge compaction is enabled

Indhumathi27 opened a new pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156


    ### Why is this PR needed?
    
    
    ### What changes were proposed in this PR?
   
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] akashrn5 commented on pull request #4156: [CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-866609542


   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] brijoobopanna commented on pull request #4156: [CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
brijoobopanna commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-866218552


   retest this please
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4156: [CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-866143841


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3864/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4156: [CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-866141328


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5609/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4156: [WIP] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-865269557


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3853/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] akashrn5 commented on a change in pull request #4156: [CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#discussion_r656150207



##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
##########
@@ -1187,6 +1187,22 @@ class UpdateCarbonTableTestCase extends QueryTest with BeforeAndAfterAll {
      Seq(Row("abc", 1, "aa", "aaa")))
   }
 
+  test("test update on table with auto-merge enabled") {
+    sql("""drop table if exists iud.autoMergeUpdate""").collect()
+    sql("""create table iud.autoMergeUpdate (c1 string,c2 int,c3 string,c5 string) STORED AS
+        |carbondata tblproperties('auto_load_merge'='true')""".stripMargin)
+    sql(s"""LOAD DATA LOCAL INPATH '$resourcesPath/IUD/dest.csv' INTO table iud.autoMergeUpdate""")
+    sql("update iud.autoMergeUpdate up_TAble set(up_table.C1)=('abc')").show()
+    sql(s"""LOAD DATA LOCAL INPATH '$resourcesPath/IUD/dest.csv' INTO table iud.autoMergeUpdate""")
+    sql("update iud.autoMergeUpdate up_TAble set(up_table.C1)=('abcd')").show()

Review comment:
       here in this test case may be can add assert for segment number by changing the compaction threshold




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] Indhumathi27 commented on pull request #4156: [WIP] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-865177264


   retest this please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4156: [WIP] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-865267122


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5596/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4156: [WIP] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-865152267






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4156: [CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-866557656


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3870/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4156: [WIP] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-865152267


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5595/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] Indhumathi27 commented on pull request #4156: [WIP] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-865177264


   retest this please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4156: [CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-866307335


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3867/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4156: [WIP] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-865152660


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3851/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] asfgit closed pull request #4156: [CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4156: [CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#discussion_r656271595



##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
##########
@@ -1187,6 +1187,22 @@ class UpdateCarbonTableTestCase extends QueryTest with BeforeAndAfterAll {
      Seq(Row("abc", 1, "aa", "aaa")))
   }
 
+  test("test update on table with auto-merge enabled") {
+    sql("""drop table if exists iud.autoMergeUpdate""").collect()
+    sql("""create table iud.autoMergeUpdate (c1 string,c2 int,c3 string,c5 string) STORED AS
+        |carbondata tblproperties('auto_load_merge'='true')""".stripMargin)
+    sql(s"""LOAD DATA LOCAL INPATH '$resourcesPath/IUD/dest.csv' INTO table iud.autoMergeUpdate""")
+    sql("update iud.autoMergeUpdate up_TAble set(up_table.C1)=('abc')").show()
+    sql(s"""LOAD DATA LOCAL INPATH '$resourcesPath/IUD/dest.csv' INTO table iud.autoMergeUpdate""")
+    sql("update iud.autoMergeUpdate up_TAble set(up_table.C1)=('abcd')").show()

Review comment:
       auto compaction does not happen without this fix. Added a assert to check the time difference




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4156: [CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-866559873


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5615/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] akashrn5 commented on pull request #4156: [CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-866503414


   retest this please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4156: [CARBONDATA-4225] Fix Update performance issues when auto merge compaction is enabled

Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4156:
URL: https://github.com/apache/carbondata/pull/4156#issuecomment-866306209


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5612/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org