You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "xumingming (via GitHub)" <gi...@apache.org> on 2024/03/19 23:29:02 UTC
[PR] [GLUTEN-5024][VL] Enhance buildbundle-veloxbe.sh to run single step [incubator-gluten]
xumingming opened a new pull request, #5032:
URL: https://github.com/apache/incubator-gluten/pull/5032
buildbundle-veloxbe.sh is composed of at least two steps:
- Get and build Velox
- Build Gluten CPP code
Currently it is not smooth to run on some platforms such as MacOS, we might need to run this script multiple times to fix a build issue, e.g. `Get and build Velox` succeed, but `Build Gluten CPP code` failed, in this PR we introduced the ability to specify the commands to run rather than run the whole script every time. By default we still run all the steps. Usage samples are the following:
```bash
-- Run all the steps
./dev/buildbundle-veloxbe.sh --build_type=RelWithDebInfo
-- Only run build_gluten_cpp
./dev/buildbundle-veloxbe.sh --build_type=RelWithDebInfo build_gluten_cpp
```
Fixes: #5024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5024][VL] Enhance buildbundle-veloxbe.sh to run single step [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #5032:
URL: https://github.com/apache/incubator-gluten/pull/5032#issuecomment-2010986896
Thanks @xumingming for the enhancement.
> can you add one flag like -Dfetch_velox=ON/OFF which can fetch velox or not before build? by default let's set it ON.
Yes that would be one of the commen case. I assume there could probably be a `fetch_velox` before
> build_velox
build_gluten_cpp
Though I am not sure if it's trivial to add. If this tricky we can apply this PR and do the further step by step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5024][VL] Enhance buildbundle-veloxbe.sh to run single step [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #5032:
URL: https://github.com/apache/incubator-gluten/pull/5032#issuecomment-2013432783
>Another issue is that currently script skip the velox build even if we modified some file in velox because it check the commit id to continue. So we need a git commit --amend in velox dir to force to build. Is there any better way to build velox automatically?
Let's fix this issue at least. It's really not good experience to run git commit --amend each time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5024][VL] Enhance buildbundle-veloxbe.sh to run single step [incubator-gluten]
Posted by "GlutenPerfBot (via GitHub)" <gi...@apache.org>.
GlutenPerfBot commented on PR #5032:
URL: https://github.com/apache/incubator-gluten/pull/5032#issuecomment-2013994909
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
<table border="1" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td><font color=#000000>query</font></td>
<td><font color=#000000>log/native_master_03_21_2024_time.csv</font></td>
<td><font color=#000000>log/native_master_03_20_2024_e1f0c01a7_time.csv</font></td>
<td><font color=#000000>difference</font></td>
<td><font color=#000000>percentage</font></td>
</tr>
<tr>
<td><font color=#F92663>q1</font></td>
<td><font color=#F92663>36.78</font></td>
<td><font color=#F92663>38.53</font></td>
<td><font color=#F92663>1.747</font></td>
<td><font color=#F92663>104.75%</font></td>
</tr>
<tr>
<td><font color=#F92663>q2</font></td>
<td><font color=#F92663>23.82</font></td>
<td><font color=#F92663>24.00</font></td>
<td><font color=#F92663>0.181</font></td>
<td><font color=#F92663>100.76%</font></td>
</tr>
<tr>
<td><font color=#F92663>q3</font></td>
<td><font color=#F92663>36.97</font></td>
<td><font color=#F92663>37.50</font></td>
<td><font color=#F92663>0.524</font></td>
<td><font color=#F92663>101.42%</font></td>
</tr>
<tr>
<td><font color=#F92663>q4</font></td>
<td><font color=#F92663>38.29</font></td>
<td><font color=#F92663>38.61</font></td>
<td><font color=#F92663>0.322</font></td>
<td><font color=#F92663>100.84%</font></td>
</tr>
<tr>
<td><font color=#F92663>q5</font></td>
<td><font color=#F92663>68.82</font></td>
<td><font color=#F92663>71.63</font></td>
<td><font color=#F92663>2.807</font></td>
<td><font color=#F92663>104.08%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q6</font></td>
<td><font color=#6F9915>7.79</font></td>
<td><font color=#6F9915>7.37</font></td>
<td><font color=#6F9915>-0.430</font></td>
<td><font color=#6F9915>94.49%</font></td>
</tr>
<tr>
<td><font color=#F92663>q7</font></td>
<td><font color=#F92663>82.16</font></td>
<td><font color=#F92663>83.96</font></td>
<td><font color=#F92663>1.799</font></td>
<td><font color=#F92663>102.19%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q8</font></td>
<td><font color=#6F9915>85.93</font></td>
<td><font color=#6F9915>84.46</font></td>
<td><font color=#6F9915>-1.468</font></td>
<td><font color=#6F9915>98.29%</font></td>
</tr>
<tr>
<td><font color=#F92663>q9</font></td>
<td><font color=#F92663>120.31</font></td>
<td><font color=#F92663>125.15</font></td>
<td><font color=#F92663>4.841</font></td>
<td><font color=#F92663>104.02%</font></td>
</tr>
<tr>
<td><font color=#F92663>q10</font></td>
<td><font color=#F92663>43.15</font></td>
<td><font color=#F92663>45.58</font></td>
<td><font color=#F92663>2.425</font></td>
<td><font color=#F92663>105.62%</font></td>
</tr>
<tr>
<td><font color=#F92663>q11</font></td>
<td><font color=#F92663>20.44</font></td>
<td><font color=#F92663>20.52</font></td>
<td><font color=#F92663>0.077</font></td>
<td><font color=#F92663>100.38%</font></td>
</tr>
<tr>
<td><font color=#F92663>q12</font></td>
<td><font color=#F92663>26.75</font></td>
<td><font color=#F92663>28.80</font></td>
<td><font color=#F92663>2.055</font></td>
<td><font color=#F92663>107.68%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q13</font></td>
<td><font color=#6F9915>48.78</font></td>
<td><font color=#6F9915>48.70</font></td>
<td><font color=#6F9915>-0.077</font></td>
<td><font color=#6F9915>99.84%</font></td>
</tr>
<tr>
<td><font color=#F92663>q14</font></td>
<td><font color=#F92663>21.16</font></td>
<td><font color=#F92663>22.25</font></td>
<td><font color=#F92663>1.086</font></td>
<td><font color=#F92663>105.13%</font></td>
</tr>
<tr>
<td><font color=#F92663>q15</font></td>
<td><font color=#F92663>28.73</font></td>
<td><font color=#F92663>31.14</font></td>
<td><font color=#F92663>2.412</font></td>
<td><font color=#F92663>108.40%</font></td>
</tr>
<tr>
<td><font color=#F92663>q16</font></td>
<td><font color=#F92663>14.42</font></td>
<td><font color=#F92663>15.42</font></td>
<td><font color=#F92663>1.002</font></td>
<td><font color=#F92663>106.95%</font></td>
</tr>
<tr>
<td><font color=#F92663>q17</font></td>
<td><font color=#F92663>99.19</font></td>
<td><font color=#F92663>102.47</font></td>
<td><font color=#F92663>3.283</font></td>
<td><font color=#F92663>103.31%</font></td>
</tr>
<tr>
<td><font color=#F92663>q18</font></td>
<td><font color=#F92663>139.46</font></td>
<td><font color=#F92663>143.60</font></td>
<td><font color=#F92663>4.134</font></td>
<td><font color=#F92663>102.96%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q19</font></td>
<td><font color=#6F9915>15.07</font></td>
<td><font color=#6F9915>13.64</font></td>
<td><font color=#6F9915>-1.429</font></td>
<td><font color=#6F9915>90.52%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q20</font></td>
<td><font color=#6F9915>27.86</font></td>
<td><font color=#6F9915>27.02</font></td>
<td><font color=#6F9915>-0.838</font></td>
<td><font color=#6F9915>96.99%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q21</font></td>
<td><font color=#6F9915>226.99</font></td>
<td><font color=#6F9915>225.52</font></td>
<td><font color=#6F9915>-1.468</font></td>
<td><font color=#6F9915>99.35%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q22</font></td>
<td><font color=#6F9915>14.13</font></td>
<td><font color=#6F9915>13.96</font></td>
<td><font color=#6F9915>-0.171</font></td>
<td><font color=#6F9915>98.79%</font></td>
</tr>
<tr>
<td><font color=#F92663>total</font></td>
<td><font color=#F92663>1226.99</font></td>
<td><font color=#F92663>1249.81</font></td>
<td><font color=#F92663>22.814</font></td>
<td><font color=#F92663>101.86%</font></td>
</tr>
</tbody>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5024][VL] Enhance buildbundle-veloxbe.sh to run single step [incubator-gluten]
Posted by "xumingming (via GitHub)" <gi...@apache.org>.
xumingming commented on PR #5032:
URL: https://github.com/apache/incubator-gluten/pull/5032#issuecomment-2010850789
@FelixYBW Would you take a look at this one? I think it saves time for build's debuging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5024][VL] Enhance buildbundle-veloxbe.sh to run single step [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #5032:
URL: https://github.com/apache/incubator-gluten/pull/5032#issuecomment-2010942825
@zhztheplayer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5024][VL] Enhance buildbundle-veloxbe.sh to run single step [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW merged PR #5032:
URL: https://github.com/apache/incubator-gluten/pull/5032
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5024][VL] Enhance buildbundle-veloxbe.sh to run single step [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5032:
URL: https://github.com/apache/incubator-gluten/pull/5032#issuecomment-2008338557
https://github.com/apache/incubator-gluten/issues/5024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5024][VL] Enhance buildbundle-veloxbe.sh to run single step [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #5032:
URL: https://github.com/apache/incubator-gluten/pull/5032#issuecomment-2010942586
Thank you. It's something we just planed. Since build_velox is always needed, can you add one flag like -Dfetch_velox=ON/OFF which can fetch velox or not before build? by default let's set it ON. We may also check if velox is fetched if not it always fetch.
Another issue is that currently script skip the velox build even if we modified some file in velox because it check the commit id to continue. So we need a git commit --amend in velox dir to force to build. Is there any better way to build velox automatically?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5024][VL] Enhance buildbundle-veloxbe.sh to run single step [incubator-gluten]
Posted by "xumingming (via GitHub)" <gi...@apache.org>.
xumingming commented on PR #5032:
URL: https://github.com/apache/incubator-gluten/pull/5032#issuecomment-2014063223
> Let's fix this issue at least. It's really not good experience to run `git commit --amend` each time
>
> @xumingming would you like to take a look?
@FelixYBW Would you create an issue for this? I'm not quite clear with what you are saying, but glad to help if you give more detailed information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org