You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "gaoyangxiaozhu (via GitHub)" <gi...@apache.org> on 2024/04/19 08:08:23 UTC
[PR] [GLUTEN-5341] [Part 2] fix fail 3.5 ut of VeloxParquetWriteSuite [incubator-gluten]
gaoyangxiaozhu opened a new pull request, #5463:
URL: https://github.com/apache/incubator-gluten/pull/5463
## What changes were proposed in this pull request?
fix test write parquet with compression codec ut of VeloxParquetWriteSuite
(Fixes: \#5341)
## How was this patch tested?
manually run
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5341] fix fail 3.5 ut of VeloxParquetWriteSuite [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5463:
URL: https://github.com/apache/incubator-gluten/pull/5463#issuecomment-2066051883
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5341] fix fail 3.5 ut of VeloxParquetWriteSuite [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #5463:
URL: https://github.com/apache/incubator-gluten/pull/5463#issuecomment-2067424505
CC @JkSelf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5341] fix fail 3.5 ut of VeloxParquetWriteSuite [incubator-gluten]
Posted by "zhli1142015 (via GitHub)" <gi...@apache.org>.
zhli1142015 merged PR #5463:
URL: https://github.com/apache/incubator-gluten/pull/5463
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5341] fix fail 3.5 ut of VeloxParquetWriteSuite [incubator-gluten]
Posted by "yma11 (via GitHub)" <gi...@apache.org>.
yma11 commented on code in PR #5463:
URL: https://github.com/apache/incubator-gluten/pull/5463#discussion_r1572008458
##########
backends-velox/src/test/scala/org/apache/spark/sql/execution/VeloxParquetWriteSuite.scala:
##########
@@ -77,7 +72,7 @@ class VeloxParquetWriteSuite extends VeloxWholeStageTransformerSuite {
val files = f.list()
assert(files.nonEmpty, extension)
- if (!SparkShimLoader.getSparkVersion.startsWith("3.4")) {
+ if (!isSparkVersionGE("3.4")) {
Review Comment:
I noticed this too, seems snappy doesn't exist, do you know the reason? here you seems you skip the check for Spark3.5?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5341] fix fail 3.5 ut of VeloxParquetWriteSuite [incubator-gluten]
Posted by "gaoyangxiaozhu (via GitHub)" <gi...@apache.org>.
gaoyangxiaozhu commented on code in PR #5463:
URL: https://github.com/apache/incubator-gluten/pull/5463#discussion_r1572184239
##########
backends-velox/src/test/scala/org/apache/spark/sql/execution/VeloxParquetWriteSuite.scala:
##########
@@ -77,7 +72,7 @@ class VeloxParquetWriteSuite extends VeloxWholeStageTransformerSuite {
val files = f.list()
assert(files.nonEmpty, extension)
- if (!SparkShimLoader.getSparkVersion.startsWith("3.4")) {
+ if (!isSparkVersionGE("3.4")) {
Review Comment:
hey @yma11 for spark 3.4+ the generated parquet file name changed
before for example in 3.3 it is
```
part-00000-1e54f8f2-0858-489c-b43f-32e6f48b3b49-c000.snappy.parquet
```
but now it is
```
Gluten_Stage_8_TID_8_0_1_e34cee0f-625d-484e-887f-c42d0f57a8d6.parquet
```
the file name not contains the codec type anymore, that's why i skip the check for this part
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5341] fix fail 3.5 ut of VeloxParquetWriteSuite [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5463:
URL: https://github.com/apache/incubator-gluten/pull/5463#issuecomment-2066051148
https://github.com/apache/incubator-gluten/issues/5341
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5341] fix fail 3.5 ut of VeloxParquetWriteSuite [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5463:
URL: https://github.com/apache/incubator-gluten/pull/5463#issuecomment-2066313673
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5341] fix fail 3.5 ut of VeloxParquetWriteSuite [incubator-gluten]
Posted by "gaoyangxiaozhu (via GitHub)" <gi...@apache.org>.
gaoyangxiaozhu commented on PR #5463:
URL: https://github.com/apache/incubator-gluten/pull/5463#issuecomment-2066839218
@yma11 / @zhouyuan for help sign
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5341] fix fail 3.5 ut of VeloxParquetWriteSuite [incubator-gluten]
Posted by "GlutenPerfBot (via GitHub)" <gi...@apache.org>.
GlutenPerfBot commented on PR #5463:
URL: https://github.com/apache/incubator-gluten/pull/5463#issuecomment-2068852615
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
<table border="1" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td><font color=#000000>query</font></td>
<td><font color=#000000>log/native_5463_time.csv</font></td>
<td><font color=#000000>log/native_master_04_21_2024_b08258e51_time.csv</font></td>
<td><font color=#000000>difference</font></td>
<td><font color=#000000>percentage</font></td>
</tr>
<tr>
<td><font color=#6F9915>q1</font></td>
<td><font color=#6F9915>36.30</font></td>
<td><font color=#6F9915>34.90</font></td>
<td><font color=#6F9915>-1.401</font></td>
<td><font color=#6F9915>96.14%</font></td>
</tr>
<tr>
<td><font color=#F92663>q2</font></td>
<td><font color=#F92663>23.90</font></td>
<td><font color=#F92663>24.06</font></td>
<td><font color=#F92663>0.160</font></td>
<td><font color=#F92663>100.67%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q3</font></td>
<td><font color=#6F9915>37.80</font></td>
<td><font color=#6F9915>36.70</font></td>
<td><font color=#6F9915>-1.104</font></td>
<td><font color=#6F9915>97.08%</font></td>
</tr>
<tr>
<td><font color=#F92663>q4</font></td>
<td><font color=#F92663>39.18</font></td>
<td><font color=#F92663>40.96</font></td>
<td><font color=#F92663>1.782</font></td>
<td><font color=#F92663>104.55%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q5</font></td>
<td><font color=#6F9915>71.82</font></td>
<td><font color=#6F9915>69.60</font></td>
<td><font color=#6F9915>-2.222</font></td>
<td><font color=#6F9915>96.91%</font></td>
</tr>
<tr>
<td><font color=#F92663>q6</font></td>
<td><font color=#F92663>7.01</font></td>
<td><font color=#F92663>7.44</font></td>
<td><font color=#F92663>0.436</font></td>
<td><font color=#F92663>106.22%</font></td>
</tr>
<tr>
<td><font color=#F92663>q7</font></td>
<td><font color=#F92663>83.42</font></td>
<td><font color=#F92663>85.59</font></td>
<td><font color=#F92663>2.167</font></td>
<td><font color=#F92663>102.60%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q8</font></td>
<td><font color=#6F9915>85.81</font></td>
<td><font color=#6F9915>85.37</font></td>
<td><font color=#6F9915>-0.436</font></td>
<td><font color=#6F9915>99.49%</font></td>
</tr>
<tr>
<td><font color=#F92663>q9</font></td>
<td><font color=#F92663>121.92</font></td>
<td><font color=#F92663>125.78</font></td>
<td><font color=#F92663>3.863</font></td>
<td><font color=#F92663>103.17%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q10</font></td>
<td><font color=#6F9915>47.42</font></td>
<td><font color=#6F9915>43.35</font></td>
<td><font color=#6F9915>-4.067</font></td>
<td><font color=#6F9915>91.42%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q11</font></td>
<td><font color=#6F9915>20.77</font></td>
<td><font color=#6F9915>19.84</font></td>
<td><font color=#6F9915>-0.928</font></td>
<td><font color=#6F9915>95.53%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q12</font></td>
<td><font color=#6F9915>27.28</font></td>
<td><font color=#6F9915>25.80</font></td>
<td><font color=#6F9915>-1.476</font></td>
<td><font color=#6F9915>94.59%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q13</font></td>
<td><font color=#6F9915>55.81</font></td>
<td><font color=#6F9915>54.10</font></td>
<td><font color=#6F9915>-1.706</font></td>
<td><font color=#6F9915>96.94%</font></td>
</tr>
<tr>
<td><font color=#F92663>q14</font></td>
<td><font color=#F92663>17.79</font></td>
<td><font color=#F92663>21.91</font></td>
<td><font color=#F92663>4.123</font></td>
<td><font color=#F92663>123.17%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q15</font></td>
<td><font color=#6F9915>29.52</font></td>
<td><font color=#6F9915>29.24</font></td>
<td><font color=#6F9915>-0.277</font></td>
<td><font color=#6F9915>99.06%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q16</font></td>
<td><font color=#6F9915>14.54</font></td>
<td><font color=#6F9915>14.28</font></td>
<td><font color=#6F9915>-0.254</font></td>
<td><font color=#6F9915>98.26%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q17</font></td>
<td><font color=#6F9915>102.46</font></td>
<td><font color=#6F9915>101.35</font></td>
<td><font color=#6F9915>-1.112</font></td>
<td><font color=#6F9915>98.91%</font></td>
</tr>
<tr>
<td><font color=#F92663>q18</font></td>
<td><font color=#F92663>141.93</font></td>
<td><font color=#F92663>145.10</font></td>
<td><font color=#F92663>3.163</font></td>
<td><font color=#F92663>102.23%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q19</font></td>
<td><font color=#6F9915>15.68</font></td>
<td><font color=#6F9915>13.61</font></td>
<td><font color=#6F9915>-2.069</font></td>
<td><font color=#6F9915>86.80%</font></td>
</tr>
<tr>
<td><font color=#F92663>q20</font></td>
<td><font color=#F92663>27.83</font></td>
<td><font color=#F92663>28.46</font></td>
<td><font color=#F92663>0.627</font></td>
<td><font color=#F92663>102.25%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q21</font></td>
<td><font color=#6F9915>296.18</font></td>
<td><font color=#6F9915>287.28</font></td>
<td><font color=#6F9915>-8.901</font></td>
<td><font color=#6F9915>96.99%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q22</font></td>
<td><font color=#6F9915>16.20</font></td>
<td><font color=#6F9915>14.78</font></td>
<td><font color=#6F9915>-1.420</font></td>
<td><font color=#6F9915>91.24%</font></td>
</tr>
<tr>
<td><font color=#6F9915>total</font></td>
<td><font color=#6F9915>1320.57</font></td>
<td><font color=#6F9915>1309.52</font></td>
<td><font color=#6F9915>-11.051</font></td>
<td><font color=#6F9915>99.16%</font></td>
</tr>
</tbody>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org