You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zzcclp (via GitHub)" <gi...@apache.org> on 2024/04/19 04:18:34 UTC
[PR] [GLUTEN-5454][CH] Support delete/update/optimize/vacuum API for the MergeTree + Delta [incubator-gluten]
zzcclp opened a new pull request, #5460:
URL: https://github.com/apache/incubator-gluten/pull/5460
## What changes were proposed in this pull request?
Support delete/update/optimize/vacuum API for the MergeTree + Delta: Similar to the DeltaTable API in Delta, add a ClickhouseTable to support delete/update/optimize/vacuum for the MergeTree + Delta by the api way:
```
// create a ClickhouseTable, the dataPath is the delta root path
val clickhouseTable = ClickhouseTable.forPath(spark, dataPath)
// optimize
clickhouseTable.optimize().executeCompaction()
// vacuum
clickhouseTable.vacuum(0.0)
// update
clickhouseTable.updateExpr("l_orderkey = 10086", Map("l_returnflag" -> "'X'"))
// delete
clickhouseTable.delete("mod(l_orderkey, 3) = 2")
```
Close #5454.
(Fixes: #5454)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5454][CH] Support delete/update/optimize/vacuum API for the MergeTree + Delta [incubator-gluten]
Posted by "GlutenPerfBot (via GitHub)" <gi...@apache.org>.
GlutenPerfBot commented on PR #5460:
URL: https://github.com/apache/incubator-gluten/pull/5460#issuecomment-2067353132
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
<table border="1" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td><font color=#000000>query</font></td>
<td><font color=#000000>log/native_master_04_19_2024_time.csv</font></td>
<td><font color=#000000>log/native_master_04_17_2024_9b3f59a1c_time.csv</font></td>
<td><font color=#000000>difference</font></td>
<td><font color=#000000>percentage</font></td>
</tr>
<tr>
<td><font color=#6F9915>q1</font></td>
<td><font color=#6F9915>36.76</font></td>
<td><font color=#6F9915>36.61</font></td>
<td><font color=#6F9915>-0.156</font></td>
<td><font color=#6F9915>99.58%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q2</font></td>
<td><font color=#6F9915>24.12</font></td>
<td><font color=#6F9915>24.12</font></td>
<td><font color=#6F9915>-0.001</font></td>
<td><font color=#6F9915>99.99%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q3</font></td>
<td><font color=#6F9915>37.49</font></td>
<td><font color=#6F9915>37.01</font></td>
<td><font color=#6F9915>-0.479</font></td>
<td><font color=#6F9915>98.72%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q4</font></td>
<td><font color=#6F9915>40.37</font></td>
<td><font color=#6F9915>38.03</font></td>
<td><font color=#6F9915>-2.339</font></td>
<td><font color=#6F9915>94.21%</font></td>
</tr>
<tr>
<td><font color=#F92663>q5</font></td>
<td><font color=#F92663>69.72</font></td>
<td><font color=#F92663>70.92</font></td>
<td><font color=#F92663>1.195</font></td>
<td><font color=#F92663>101.71%</font></td>
</tr>
<tr>
<td><font color=#F92663>q6</font></td>
<td><font color=#F92663>5.81</font></td>
<td><font color=#F92663>5.81</font></td>
<td><font color=#F92663>0.000</font></td>
<td><font color=#F92663>100.00%</font></td>
</tr>
<tr>
<td><font color=#F92663>q7</font></td>
<td><font color=#F92663>83.27</font></td>
<td><font color=#F92663>86.12</font></td>
<td><font color=#F92663>2.846</font></td>
<td><font color=#F92663>103.42%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q8</font></td>
<td><font color=#6F9915>84.49</font></td>
<td><font color=#6F9915>82.86</font></td>
<td><font color=#6F9915>-1.633</font></td>
<td><font color=#6F9915>98.07%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q9</font></td>
<td><font color=#6F9915>124.94</font></td>
<td><font color=#6F9915>123.45</font></td>
<td><font color=#6F9915>-1.483</font></td>
<td><font color=#6F9915>98.81%</font></td>
</tr>
<tr>
<td><font color=#F92663>q10</font></td>
<td><font color=#F92663>44.29</font></td>
<td><font color=#F92663>45.70</font></td>
<td><font color=#F92663>1.410</font></td>
<td><font color=#F92663>103.18%</font></td>
</tr>
<tr>
<td><font color=#F92663>q11</font></td>
<td><font color=#F92663>20.07</font></td>
<td><font color=#F92663>20.35</font></td>
<td><font color=#F92663>0.275</font></td>
<td><font color=#F92663>101.37%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q12</font></td>
<td><font color=#6F9915>31.25</font></td>
<td><font color=#6F9915>28.15</font></td>
<td><font color=#6F9915>-3.108</font></td>
<td><font color=#6F9915>90.05%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q13</font></td>
<td><font color=#6F9915>55.12</font></td>
<td><font color=#6F9915>54.46</font></td>
<td><font color=#6F9915>-0.658</font></td>
<td><font color=#6F9915>98.81%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q14</font></td>
<td><font color=#6F9915>18.53</font></td>
<td><font color=#6F9915>17.88</font></td>
<td><font color=#6F9915>-0.649</font></td>
<td><font color=#6F9915>96.50%</font></td>
</tr>
<tr>
<td><font color=#F92663>q15</font></td>
<td><font color=#F92663>30.13</font></td>
<td><font color=#F92663>30.69</font></td>
<td><font color=#F92663>0.562</font></td>
<td><font color=#F92663>101.87%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q16</font></td>
<td><font color=#6F9915>14.22</font></td>
<td><font color=#6F9915>14.00</font></td>
<td><font color=#6F9915>-0.217</font></td>
<td><font color=#6F9915>98.47%</font></td>
</tr>
<tr>
<td><font color=#F92663>q17</font></td>
<td><font color=#F92663>100.91</font></td>
<td><font color=#F92663>101.27</font></td>
<td><font color=#F92663>0.360</font></td>
<td><font color=#F92663>100.36%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q18</font></td>
<td><font color=#6F9915>144.19</font></td>
<td><font color=#6F9915>143.83</font></td>
<td><font color=#6F9915>-0.367</font></td>
<td><font color=#6F9915>99.75%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q19</font></td>
<td><font color=#6F9915>14.30</font></td>
<td><font color=#6F9915>13.61</font></td>
<td><font color=#6F9915>-0.686</font></td>
<td><font color=#6F9915>95.20%</font></td>
</tr>
<tr>
<td><font color=#F92663>q20</font></td>
<td><font color=#F92663>27.41</font></td>
<td><font color=#F92663>27.86</font></td>
<td><font color=#F92663>0.457</font></td>
<td><font color=#F92663>101.67%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q21</font></td>
<td><font color=#6F9915>293.61</font></td>
<td><font color=#6F9915>285.86</font></td>
<td><font color=#6F9915>-7.747</font></td>
<td><font color=#6F9915>97.36%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q22</font></td>
<td><font color=#6F9915>15.15</font></td>
<td><font color=#6F9915>14.42</font></td>
<td><font color=#6F9915>-0.732</font></td>
<td><font color=#6F9915>95.17%</font></td>
</tr>
<tr>
<td><font color=#6F9915>total</font></td>
<td><font color=#6F9915>1316.16</font></td>
<td><font color=#6F9915>1303.01</font></td>
<td><font color=#6F9915>-13.150</font></td>
<td><font color=#6F9915>99.00%</font></td>
</tr>
</tbody>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5454][CH] Support delete/update/optimize/vacuum API for the MergeTree + Delta [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5460:
URL: https://github.com/apache/incubator-gluten/pull/5460#issuecomment-2065725064
https://github.com/apache/incubator-gluten/issues/5454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5454][CH] Support delete/update/optimize/vacuum API for the MergeTree + Delta [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5460:
URL: https://github.com/apache/incubator-gluten/pull/5460#issuecomment-2065725808
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5454][CH] Support delete/update/optimize/vacuum API for the MergeTree + Delta [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen merged PR #5460:
URL: https://github.com/apache/incubator-gluten/pull/5460
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5454][CH] Support delete/update/optimize/vacuum API for the MergeTree + Delta [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5460:
URL: https://github.com/apache/incubator-gluten/pull/5460#issuecomment-2065844112
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5454][CH] Support delete/update/optimize/vacuum API for the MergeTree + Delta [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5460:
URL: https://github.com/apache/incubator-gluten/pull/5460#issuecomment-2065841565
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5454][CH] Support delete/update/optimize/vacuum API for the MergeTree + Delta [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5460:
URL: https://github.com/apache/incubator-gluten/pull/5460#issuecomment-2065848934
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5454][CH] Support delete/update/optimize/vacuum API for the MergeTree + Delta [incubator-gluten]
Posted by "loneylee (via GitHub)" <gi...@apache.org>.
loneylee commented on PR #5460:
URL: https://github.com/apache/incubator-gluten/pull/5460#issuecomment-2066148253
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org