You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "jinchengchenghh (via GitHub)" <gi...@apache.org> on 2024/08/21 09:05:22 UTC
[PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
jinchengchenghh opened a new pull request, #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961][VL][feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#issuecomment-2316630833
> Anymore comments? @zhztheplayer
Just some nits, I updated the PR to save some time. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#discussion_r1726065033
##########
gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java:
##########
@@ -1255,9 +1266,8 @@ void setNull(int rowId) {
throw new UnsupportedOperationException();
}
- void setNotNull(int rowId) {
- throw new UnsupportedOperationException();
- }
+ // Arrow not need to setNotNull, set the valus is enough.
+ void setNotNull(int rowId) {}
Review Comment:
Any reason about the change?
##########
gluten-data/src/main/scala/org/apache/spark/sql/utils/SparkSchemaUtil.scala:
##########
@@ -37,6 +37,16 @@ object SparkSchemaUtil {
SparkArrowUtil.toArrowSchema(schema, timeZoneId)
}
+ def checkSchema(schema: StructType): Boolean = {
+ try {
+ SparkSchemaUtil.toArrowSchema(schema)
+ true
+ } catch {
+ case _: Exception =>
+ false
+ }
+ }
Review Comment:
What's the error you have seen then decide to catch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#issuecomment-2314350557
Anymore comments? @zhztheplayer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961][VL][feat] Add decimal write support for ArrowWritableColumnVector [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer merged PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#discussion_r1726160287
##########
gluten-data/src/main/scala/org/apache/spark/sql/utils/SparkSchemaUtil.scala:
##########
@@ -37,6 +37,16 @@ object SparkSchemaUtil {
SparkArrowUtil.toArrowSchema(schema, timeZoneId)
}
+ def checkSchema(schema: StructType): Boolean = {
+ try {
+ SparkSchemaUtil.toArrowSchema(schema)
+ true
+ } catch {
+ case _: Exception =>
+ false
+ }
+ }
Review Comment:
I find here will throw exception, https://github.com/apache/incubator-gluten/blob/main/gluten-data/src/main/scala/org/apache/spark/sql/utils/SparkArrowUtil.scala#L54
But it will not run to here actually, I will drop the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#discussion_r1726182070
##########
gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java:
##########
@@ -1255,9 +1266,8 @@ void setNull(int rowId) {
throw new UnsupportedOperationException();
}
- void setNotNull(int rowId) {
- throw new UnsupportedOperationException();
- }
+ // Arrow not need to setNotNull, set the valus is enough.
+ void setNotNull(int rowId) {}
Review Comment:
The class should not be abstract since its subclass will not implement the functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#issuecomment-2301544581
https://github.com/apache/incubator-gluten/issues/6961
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#discussion_r1728190880
##########
gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java:
##########
@@ -1255,9 +1266,8 @@ void setNull(int rowId) {
throw new UnsupportedOperationException();
}
- void setNotNull(int rowId) {
- throw new UnsupportedOperationException();
- }
+ // Arrow not need to setNotNull, set the valus is enough.
+ void setNotNull(int rowId) {}
Review Comment:
I will override putByteArray but check if it put all bytes, I will refactor it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#discussion_r1726075975
##########
gluten-data/src/main/scala/org/apache/spark/sql/utils/SparkSchemaUtil.scala:
##########
@@ -37,6 +37,16 @@ object SparkSchemaUtil {
SparkArrowUtil.toArrowSchema(schema, timeZoneId)
}
+ def checkSchema(schema: StructType): Boolean = {
+ try {
+ SparkSchemaUtil.toArrowSchema(schema)
+ true
+ } catch {
+ case _: Exception =>
+ false
+ }
+ }
Review Comment:
What's the error here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#discussion_r1726065033
##########
gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java:
##########
@@ -1255,9 +1266,8 @@ void setNull(int rowId) {
throw new UnsupportedOperationException();
}
- void setNotNull(int rowId) {
- throw new UnsupportedOperationException();
- }
+ // Arrow not need to setNotNull, set the valus is enough.
+ void setNotNull(int rowId) {}
Review Comment:
Any reason of the change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#discussion_r1726170128
##########
gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java:
##########
@@ -1255,9 +1266,8 @@ void setNull(int rowId) {
throw new UnsupportedOperationException();
}
- void setNotNull(int rowId) {
- throw new UnsupportedOperationException();
- }
+ // Arrow not need to setNotNull, set the valus is enough.
+ void setNotNull(int rowId) {}
Review Comment:
The class is abstract and the method should be overridable by impl classes.
See https://github.com/apache/incubator-gluten/blob/ed3d333c7c357e95f6736f378c972b714bdfb420/gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java#L1913 as example.
And abstract class `ArrowVectorWriter` doesn't likely have to be abstract. Could make it an interface as it almost doesn't override anything from the hierarchy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#discussion_r1726162605
##########
gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java:
##########
@@ -1255,9 +1266,8 @@ void setNull(int rowId) {
throw new UnsupportedOperationException();
}
- void setNotNull(int rowId) {
- throw new UnsupportedOperationException();
- }
+ // Arrow not need to setNotNull, set the valus is enough.
+ void setNotNull(int rowId) {}
Review Comment:
If we use `MutableProjection`, it will call `MutableColumnarBatchRow`, we can construct `MutableColumnarBatchRow` by `ArrowWritableColumnVector`. https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/MutableColumnarRow.java#L286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961][VL][feat] Add decimal write support for ArrowWritableColumnVector [incubator-gluten]
Posted by "GlutenPerfBot (via GitHub)" <gi...@apache.org>.
GlutenPerfBot commented on PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#issuecomment-2318656925
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
<table border="1" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td><font color=#000000>query</font></td>
<td><font color=#000000>log/native_master_08_29_2024_time.csv</font></td>
<td><font color=#000000>log/native_master_08_28_2024_3928dc2526_time.csv</font></td>
<td><font color=#000000>difference</font></td>
<td><font color=#000000>percentage</font></td>
</tr>
<tr>
<td><font color=#F92663>q1</font></td>
<td><font color=#F92663>39.06</font></td>
<td><font color=#F92663>40.45</font></td>
<td><font color=#F92663>1.391</font></td>
<td><font color=#F92663>103.56%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q2</font></td>
<td><font color=#6F9915>30.61</font></td>
<td><font color=#6F9915>30.31</font></td>
<td><font color=#6F9915>-0.301</font></td>
<td><font color=#6F9915>99.02%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q3</font></td>
<td><font color=#6F9915>53.88</font></td>
<td><font color=#6F9915>53.48</font></td>
<td><font color=#6F9915>-0.396</font></td>
<td><font color=#6F9915>99.26%</font></td>
</tr>
<tr>
<td><font color=#F92663>q4</font></td>
<td><font color=#F92663>41.59</font></td>
<td><font color=#F92663>41.72</font></td>
<td><font color=#F92663>0.130</font></td>
<td><font color=#F92663>100.31%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q5</font></td>
<td><font color=#6F9915>104.66</font></td>
<td><font color=#6F9915>102.35</font></td>
<td><font color=#6F9915>-2.314</font></td>
<td><font color=#6F9915>97.79%</font></td>
</tr>
<tr>
<td><font color=#F92663>q6</font></td>
<td><font color=#F92663>11.08</font></td>
<td><font color=#F92663>12.91</font></td>
<td><font color=#F92663>1.836</font></td>
<td><font color=#F92663>116.57%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q7</font></td>
<td><font color=#6F9915>118.69</font></td>
<td><font color=#6F9915>116.14</font></td>
<td><font color=#6F9915>-2.552</font></td>
<td><font color=#6F9915>97.85%</font></td>
</tr>
<tr>
<td><font color=#F92663>q8</font></td>
<td><font color=#F92663>115.51</font></td>
<td><font color=#F92663>117.06</font></td>
<td><font color=#F92663>1.556</font></td>
<td><font color=#F92663>101.35%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q9</font></td>
<td><font color=#6F9915>168.74</font></td>
<td><font color=#6F9915>168.17</font></td>
<td><font color=#6F9915>-0.565</font></td>
<td><font color=#6F9915>99.67%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q10</font></td>
<td><font color=#6F9915>64.93</font></td>
<td><font color=#6F9915>63.46</font></td>
<td><font color=#6F9915>-1.470</font></td>
<td><font color=#6F9915>97.74%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q11</font></td>
<td><font color=#6F9915>27.73</font></td>
<td><font color=#6F9915>26.85</font></td>
<td><font color=#6F9915>-0.881</font></td>
<td><font color=#6F9915>96.82%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q12</font></td>
<td><font color=#6F9915>31.17</font></td>
<td><font color=#6F9915>30.31</font></td>
<td><font color=#6F9915>-0.860</font></td>
<td><font color=#6F9915>97.24%</font></td>
</tr>
<tr>
<td><font color=#F92663>q13</font></td>
<td><font color=#F92663>52.16</font></td>
<td><font color=#F92663>52.28</font></td>
<td><font color=#F92663>0.119</font></td>
<td><font color=#F92663>100.23%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q14</font></td>
<td><font color=#6F9915>27.25</font></td>
<td><font color=#6F9915>25.88</font></td>
<td><font color=#6F9915>-1.378</font></td>
<td><font color=#6F9915>94.95%</font></td>
</tr>
<tr>
<td><font color=#F92663>q15</font></td>
<td><font color=#F92663>54.32</font></td>
<td><font color=#F92663>56.08</font></td>
<td><font color=#F92663>1.760</font></td>
<td><font color=#F92663>103.24%</font></td>
</tr>
<tr>
<td><font color=#F92663>q16</font></td>
<td><font color=#F92663>18.16</font></td>
<td><font color=#F92663>18.31</font></td>
<td><font color=#F92663>0.153</font></td>
<td><font color=#F92663>100.84%</font></td>
</tr>
<tr>
<td><font color=#F92663>q17</font></td>
<td><font color=#F92663>130.29</font></td>
<td><font color=#F92663>133.77</font></td>
<td><font color=#F92663>3.483</font></td>
<td><font color=#F92663>102.67%</font></td>
</tr>
<tr>
<td><font color=#F92663>q18</font></td>
<td><font color=#F92663>199.00</font></td>
<td><font color=#F92663>199.77</font></td>
<td><font color=#F92663>0.763</font></td>
<td><font color=#F92663>100.38%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q19</font></td>
<td><font color=#6F9915>30.88</font></td>
<td><font color=#6F9915>24.78</font></td>
<td><font color=#6F9915>-6.099</font></td>
<td><font color=#6F9915>80.25%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q20</font></td>
<td><font color=#6F9915>43.14</font></td>
<td><font color=#6F9915>42.75</font></td>
<td><font color=#6F9915>-0.396</font></td>
<td><font color=#6F9915>99.08%</font></td>
</tr>
<tr>
<td><font color=#F92663>q21</font></td>
<td><font color=#F92663>387.80</font></td>
<td><font color=#F92663>390.58</font></td>
<td><font color=#F92663>2.778</font></td>
<td><font color=#F92663>100.72%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q22</font></td>
<td><font color=#6F9915>17.22</font></td>
<td><font color=#6F9915>15.25</font></td>
<td><font color=#6F9915>-1.977</font></td>
<td><font color=#6F9915>88.52%</font></td>
</tr>
<tr>
<td><font color=#6F9915>total</font></td>
<td><font color=#6F9915>1767.88</font></td>
<td><font color=#6F9915>1762.66</font></td>
<td><font color=#6F9915>-5.220</font></td>
<td><font color=#6F9915>99.70%</font></td>
</tr>
</tbody>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#discussion_r1728140430
##########
gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java:
##########
@@ -1255,9 +1266,8 @@ void setNull(int rowId) {
throw new UnsupportedOperationException();
}
- void setNotNull(int rowId) {
- throw new UnsupportedOperationException();
- }
+ // Arrow not need to setNotNull, set the valus is enough.
+ void setNotNull(int rowId) {}
Review Comment:
Can we route `putDecimal` calls to `accessor.setDecimal`? I am not that familiar with the design here but looks to be a manner we should follow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#discussion_r1728180120
##########
gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java:
##########
@@ -1255,9 +1266,8 @@ void setNull(int rowId) {
throw new UnsupportedOperationException();
}
- void setNotNull(int rowId) {
- throw new UnsupportedOperationException();
- }
+ // Arrow not need to setNotNull, set the valus is enough.
+ void setNotNull(int rowId) {}
Review Comment:
writer does not have the `setDecimal` function, I copied most code from `https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java#L429` but is different in putByteArray, because Arrow DecimalVector does not support put byte array range.
Put the all bytes is final function in Spark.
```
public abstract int putByteArray(int rowId, byte[] value, int offset, int count);
public final int putByteArray(int rowId, byte[] value) {
return putByteArray(rowId, value, 0, value.length);
}
```
I will add a test for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961][VL][feat] Add decimal write support for ArrowWritableColumnVector [incubator-gluten]
Posted by "GlutenPerfBot (via GitHub)" <gi...@apache.org>.
GlutenPerfBot commented on PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#issuecomment-2318532950
===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====
<table border="1" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td><font color=#000000>query</font></td>
<td><font color=#000000>log/native_master_08_29_2024_time.csv</font></td>
<td><font color=#000000>log/native_master_08_28_2024_3928dc2526_time.csv</font></td>
<td><font color=#000000>difference</font></td>
<td><font color=#000000>percentage</font></td>
</tr>
<tr>
<td><font color=#F92663>q1</font></td>
<td><font color=#F92663>13.88</font></td>
<td><font color=#F92663>14.02</font></td>
<td><font color=#F92663>0.140</font></td>
<td><font color=#F92663>101.01%</font></td>
</tr>
<tr>
<td><font color=#F92663>q2</font></td>
<td><font color=#F92663>13.25</font></td>
<td><font color=#F92663>13.32</font></td>
<td><font color=#F92663>0.077</font></td>
<td><font color=#F92663>100.58%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q3</font></td>
<td><font color=#6F9915>4.36</font></td>
<td><font color=#6F9915>3.86</font></td>
<td><font color=#6F9915>-0.491</font></td>
<td><font color=#6F9915>88.73%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q4</font></td>
<td><font color=#6F9915>72.35</font></td>
<td><font color=#6F9915>72.05</font></td>
<td><font color=#6F9915>-0.301</font></td>
<td><font color=#6F9915>99.58%</font></td>
</tr>
<tr>
<td><font color=#F92663>q5</font></td>
<td><font color=#F92663>8.05</font></td>
<td><font color=#F92663>9.26</font></td>
<td><font color=#F92663>1.213</font></td>
<td><font color=#F92663>115.07%</font></td>
</tr>
<tr>
<td><font color=#F92663>q6</font></td>
<td><font color=#F92663>2.25</font></td>
<td><font color=#F92663>2.36</font></td>
<td><font color=#F92663>0.110</font></td>
<td><font color=#F92663>104.91%</font></td>
</tr>
<tr>
<td><font color=#F92663>q7</font></td>
<td><font color=#F92663>6.53</font></td>
<td><font color=#F92663>6.72</font></td>
<td><font color=#F92663>0.188</font></td>
<td><font color=#F92663>102.88%</font></td>
</tr>
<tr>
<td><font color=#F92663>q8</font></td>
<td><font color=#F92663>4.87</font></td>
<td><font color=#F92663>5.08</font></td>
<td><font color=#F92663>0.215</font></td>
<td><font color=#F92663>104.43%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q9</font></td>
<td><font color=#6F9915>24.26</font></td>
<td><font color=#6F9915>24.18</font></td>
<td><font color=#6F9915>-0.079</font></td>
<td><font color=#6F9915>99.67%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q10</font></td>
<td><font color=#6F9915>10.56</font></td>
<td><font color=#6F9915>9.64</font></td>
<td><font color=#6F9915>-0.918</font></td>
<td><font color=#6F9915>91.30%</font></td>
</tr>
<tr>
<td><font color=#F92663>q11</font></td>
<td><font color=#F92663>39.02</font></td>
<td><font color=#F92663>39.48</font></td>
<td><font color=#F92663>0.460</font></td>
<td><font color=#F92663>101.18%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q12</font></td>
<td><font color=#6F9915>1.53</font></td>
<td><font color=#6F9915>1.40</font></td>
<td><font color=#6F9915>-0.134</font></td>
<td><font color=#6F9915>91.24%</font></td>
</tr>
<tr>
<td><font color=#F92663>q13</font></td>
<td><font color=#F92663>6.50</font></td>
<td><font color=#F92663>6.61</font></td>
<td><font color=#F92663>0.118</font></td>
<td><font color=#F92663>101.81%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q14a</font></td>
<td><font color=#6F9915>47.24</font></td>
<td><font color=#6F9915>46.08</font></td>
<td><font color=#6F9915>-1.162</font></td>
<td><font color=#6F9915>97.54%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q14b</font></td>
<td><font color=#6F9915>43.24</font></td>
<td><font color=#6F9915>42.22</font></td>
<td><font color=#6F9915>-1.020</font></td>
<td><font color=#6F9915>97.64%</font></td>
</tr>
<tr>
<td><font color=#F92663>q15</font></td>
<td><font color=#F92663>2.78</font></td>
<td><font color=#F92663>2.78</font></td>
<td><font color=#F92663>0.006</font></td>
<td><font color=#F92663>100.23%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q16</font></td>
<td><font color=#6F9915>46.45</font></td>
<td><font color=#6F9915>46.08</font></td>
<td><font color=#6F9915>-0.371</font></td>
<td><font color=#6F9915>99.20%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q17</font></td>
<td><font color=#6F9915>5.50</font></td>
<td><font color=#6F9915>5.39</font></td>
<td><font color=#6F9915>-0.111</font></td>
<td><font color=#6F9915>97.99%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q18</font></td>
<td><font color=#6F9915>8.00</font></td>
<td><font color=#6F9915>6.91</font></td>
<td><font color=#6F9915>-1.090</font></td>
<td><font color=#6F9915>86.37%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q19</font></td>
<td><font color=#6F9915>2.42</font></td>
<td><font color=#6F9915>2.11</font></td>
<td><font color=#6F9915>-0.310</font></td>
<td><font color=#6F9915>87.22%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q20</font></td>
<td><font color=#6F9915>1.60</font></td>
<td><font color=#6F9915>1.51</font></td>
<td><font color=#6F9915>-0.089</font></td>
<td><font color=#6F9915>94.46%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q21</font></td>
<td><font color=#6F9915>1.24</font></td>
<td><font color=#6F9915>1.10</font></td>
<td><font color=#6F9915>-0.140</font></td>
<td><font color=#6F9915>88.66%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q22</font></td>
<td><font color=#6F9915>7.98</font></td>
<td><font color=#6F9915>7.71</font></td>
<td><font color=#6F9915>-0.275</font></td>
<td><font color=#6F9915>96.56%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q23a</font></td>
<td><font color=#6F9915>104.75</font></td>
<td><font color=#6F9915>103.66</font></td>
<td><font color=#6F9915>-1.089</font></td>
<td><font color=#6F9915>98.96%</font></td>
</tr>
<tr>
<td><font color=#F92663>q23b</font></td>
<td><font color=#F92663>128.72</font></td>
<td><font color=#F92663>128.88</font></td>
<td><font color=#F92663>0.164</font></td>
<td><font color=#F92663>100.13%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q24a</font></td>
<td><font color=#6F9915>111.28</font></td>
<td><font color=#6F9915>109.30</font></td>
<td><font color=#6F9915>-1.977</font></td>
<td><font color=#6F9915>98.22%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q24b</font></td>
<td><font color=#6F9915>111.05</font></td>
<td><font color=#6F9915>107.88</font></td>
<td><font color=#6F9915>-3.171</font></td>
<td><font color=#6F9915>97.14%</font></td>
</tr>
<tr>
<td><font color=#F92663>q25</font></td>
<td><font color=#F92663>4.11</font></td>
<td><font color=#F92663>4.23</font></td>
<td><font color=#F92663>0.114</font></td>
<td><font color=#F92663>102.77%</font></td>
</tr>
<tr>
<td><font color=#F92663>q26</font></td>
<td><font color=#F92663>3.02</font></td>
<td><font color=#F92663>3.24</font></td>
<td><font color=#F92663>0.221</font></td>
<td><font color=#F92663>107.31%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q27</font></td>
<td><font color=#6F9915>4.20</font></td>
<td><font color=#6F9915>4.02</font></td>
<td><font color=#6F9915>-0.185</font></td>
<td><font color=#6F9915>95.60%</font></td>
</tr>
<tr>
<td><font color=#F92663>q28</font></td>
<td><font color=#F92663>33.64</font></td>
<td><font color=#F92663>33.79</font></td>
<td><font color=#F92663>0.153</font></td>
<td><font color=#F92663>100.46%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q29</font></td>
<td><font color=#6F9915>11.19</font></td>
<td><font color=#6F9915>11.05</font></td>
<td><font color=#6F9915>-0.139</font></td>
<td><font color=#6F9915>98.75%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q30</font></td>
<td><font color=#6F9915>5.93</font></td>
<td><font color=#6F9915>4.94</font></td>
<td><font color=#6F9915>-0.993</font></td>
<td><font color=#6F9915>83.26%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q31</font></td>
<td><font color=#6F9915>7.94</font></td>
<td><font color=#6F9915>7.50</font></td>
<td><font color=#6F9915>-0.440</font></td>
<td><font color=#6F9915>94.46%</font></td>
</tr>
<tr>
<td><font color=#F92663>q32</font></td>
<td><font color=#F92663>1.17</font></td>
<td><font color=#F92663>1.45</font></td>
<td><font color=#F92663>0.285</font></td>
<td><font color=#F92663>124.40%</font></td>
</tr>
<tr>
<td><font color=#F92663>q33</font></td>
<td><font color=#F92663>4.30</font></td>
<td><font color=#F92663>4.51</font></td>
<td><font color=#F92663>0.212</font></td>
<td><font color=#F92663>104.93%</font></td>
</tr>
<tr>
<td><font color=#F92663>q34</font></td>
<td><font color=#F92663>3.81</font></td>
<td><font color=#F92663>4.11</font></td>
<td><font color=#F92663>0.301</font></td>
<td><font color=#F92663>107.91%</font></td>
</tr>
<tr>
<td><font color=#F92663>q35</font></td>
<td><font color=#F92663>7.89</font></td>
<td><font color=#F92663>8.32</font></td>
<td><font color=#F92663>0.432</font></td>
<td><font color=#F92663>105.48%</font></td>
</tr>
<tr>
<td><font color=#F92663>q36</font></td>
<td><font color=#F92663>4.73</font></td>
<td><font color=#F92663>4.74</font></td>
<td><font color=#F92663>0.010</font></td>
<td><font color=#F92663>100.22%</font></td>
</tr>
<tr>
<td><font color=#F92663>q37</font></td>
<td><font color=#F92663>4.73</font></td>
<td><font color=#F92663>5.87</font></td>
<td><font color=#F92663>1.142</font></td>
<td><font color=#F92663>124.15%</font></td>
</tr>
<tr>
<td><font color=#F92663>q38</font></td>
<td><font color=#F92663>13.59</font></td>
<td><font color=#F92663>14.50</font></td>
<td><font color=#F92663>0.911</font></td>
<td><font color=#F92663>106.70%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q39a</font></td>
<td><font color=#6F9915>3.44</font></td>
<td><font color=#6F9915>3.27</font></td>
<td><font color=#6F9915>-0.166</font></td>
<td><font color=#6F9915>95.17%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q39b</font></td>
<td><font color=#6F9915>3.06</font></td>
<td><font color=#6F9915>2.83</font></td>
<td><font color=#6F9915>-0.225</font></td>
<td><font color=#6F9915>92.65%</font></td>
</tr>
<tr>
<td><font color=#F92663>q40</font></td>
<td><font color=#F92663>3.84</font></td>
<td><font color=#F92663>3.95</font></td>
<td><font color=#F92663>0.107</font></td>
<td><font color=#F92663>102.79%</font></td>
</tr>
<tr>
<td><font color=#F92663>q41</font></td>
<td><font color=#F92663>0.64</font></td>
<td><font color=#F92663>0.70</font></td>
<td><font color=#F92663>0.062</font></td>
<td><font color=#F92663>109.68%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q42</font></td>
<td><font color=#6F9915>1.02</font></td>
<td><font color=#6F9915>0.97</font></td>
<td><font color=#6F9915>-0.056</font></td>
<td><font color=#6F9915>94.56%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q43</font></td>
<td><font color=#6F9915>4.84</font></td>
<td><font color=#6F9915>4.83</font></td>
<td><font color=#6F9915>-0.003</font></td>
<td><font color=#6F9915>99.93%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q44</font></td>
<td><font color=#6F9915>9.99</font></td>
<td><font color=#6F9915>9.98</font></td>
<td><font color=#6F9915>-0.011</font></td>
<td><font color=#6F9915>99.89%</font></td>
</tr>
<tr>
<td><font color=#F92663>q45</font></td>
<td><font color=#F92663>3.23</font></td>
<td><font color=#F92663>3.28</font></td>
<td><font color=#F92663>0.053</font></td>
<td><font color=#F92663>101.63%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q46</font></td>
<td><font color=#6F9915>3.82</font></td>
<td><font color=#6F9915>3.74</font></td>
<td><font color=#6F9915>-0.081</font></td>
<td><font color=#6F9915>97.89%</font></td>
</tr>
<tr>
<td><font color=#F92663>q47</font></td>
<td><font color=#F92663>18.93</font></td>
<td><font color=#F92663>19.36</font></td>
<td><font color=#F92663>0.426</font></td>
<td><font color=#F92663>102.25%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q48</font></td>
<td><font color=#6F9915>5.46</font></td>
<td><font color=#6F9915>5.37</font></td>
<td><font color=#6F9915>-0.092</font></td>
<td><font color=#6F9915>98.32%</font></td>
</tr>
<tr>
<td><font color=#F92663>q49</font></td>
<td><font color=#F92663>9.03</font></td>
<td><font color=#F92663>9.27</font></td>
<td><font color=#F92663>0.243</font></td>
<td><font color=#F92663>102.69%</font></td>
</tr>
<tr>
<td><font color=#F92663>q50</font></td>
<td><font color=#F92663>21.58</font></td>
<td><font color=#F92663>21.79</font></td>
<td><font color=#F92663>0.207</font></td>
<td><font color=#F92663>100.96%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q51</font></td>
<td><font color=#6F9915>9.97</font></td>
<td><font color=#6F9915>9.73</font></td>
<td><font color=#6F9915>-0.237</font></td>
<td><font color=#6F9915>97.63%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q52</font></td>
<td><font color=#6F9915>1.13</font></td>
<td><font color=#6F9915>1.06</font></td>
<td><font color=#6F9915>-0.071</font></td>
<td><font color=#6F9915>93.69%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q53</font></td>
<td><font color=#6F9915>2.44</font></td>
<td><font color=#6F9915>2.44</font></td>
<td><font color=#6F9915>-0.005</font></td>
<td><font color=#6F9915>99.78%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q54</font></td>
<td><font color=#6F9915>3.99</font></td>
<td><font color=#6F9915>3.98</font></td>
<td><font color=#6F9915>-0.012</font></td>
<td><font color=#6F9915>99.70%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q55</font></td>
<td><font color=#6F9915>1.10</font></td>
<td><font color=#6F9915>1.10</font></td>
<td><font color=#6F9915>-0.002</font></td>
<td><font color=#6F9915>99.81%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q56</font></td>
<td><font color=#6F9915>4.35</font></td>
<td><font color=#6F9915>4.15</font></td>
<td><font color=#6F9915>-0.193</font></td>
<td><font color=#6F9915>95.55%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q57</font></td>
<td><font color=#6F9915>11.07</font></td>
<td><font color=#6F9915>10.86</font></td>
<td><font color=#6F9915>-0.205</font></td>
<td><font color=#6F9915>98.15%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q58</font></td>
<td><font color=#6F9915>2.51</font></td>
<td><font color=#6F9915>2.39</font></td>
<td><font color=#6F9915>-0.118</font></td>
<td><font color=#6F9915>95.31%</font></td>
</tr>
<tr>
<td><font color=#F92663>q59</font></td>
<td><font color=#F92663>11.12</font></td>
<td><font color=#F92663>11.44</font></td>
<td><font color=#F92663>0.323</font></td>
<td><font color=#F92663>102.90%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q60</font></td>
<td><font color=#6F9915>4.10</font></td>
<td><font color=#6F9915>4.10</font></td>
<td><font color=#6F9915>-0.002</font></td>
<td><font color=#6F9915>99.96%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q61</font></td>
<td><font color=#6F9915>4.19</font></td>
<td><font color=#6F9915>4.12</font></td>
<td><font color=#6F9915>-0.063</font></td>
<td><font color=#6F9915>98.50%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q62</font></td>
<td><font color=#6F9915>4.89</font></td>
<td><font color=#6F9915>4.59</font></td>
<td><font color=#6F9915>-0.301</font></td>
<td><font color=#6F9915>93.84%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q63</font></td>
<td><font color=#6F9915>2.50</font></td>
<td><font color=#6F9915>2.35</font></td>
<td><font color=#6F9915>-0.153</font></td>
<td><font color=#6F9915>93.90%</font></td>
</tr>
<tr>
<td><font color=#F92663>q64</font></td>
<td><font color=#F92663>60.58</font></td>
<td><font color=#F92663>62.31</font></td>
<td><font color=#F92663>1.723</font></td>
<td><font color=#F92663>102.84%</font></td>
</tr>
<tr>
<td><font color=#F92663>q65</font></td>
<td><font color=#F92663>17.29</font></td>
<td><font color=#F92663>17.60</font></td>
<td><font color=#F92663>0.314</font></td>
<td><font color=#F92663>101.82%</font></td>
</tr>
<tr>
<td><font color=#F92663>q66</font></td>
<td><font color=#F92663>4.06</font></td>
<td><font color=#F92663>5.74</font></td>
<td><font color=#F92663>1.672</font></td>
<td><font color=#F92663>141.17%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q67</font></td>
<td><font color=#6F9915>408.72</font></td>
<td><font color=#6F9915>385.75</font></td>
<td><font color=#6F9915>-22.971</font></td>
<td><font color=#6F9915>94.38%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q68</font></td>
<td><font color=#6F9915>3.79</font></td>
<td><font color=#6F9915>3.55</font></td>
<td><font color=#6F9915>-0.242</font></td>
<td><font color=#6F9915>93.62%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q69</font></td>
<td><font color=#6F9915>5.56</font></td>
<td><font color=#6F9915>5.32</font></td>
<td><font color=#6F9915>-0.241</font></td>
<td><font color=#6F9915>95.66%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q70</font></td>
<td><font color=#6F9915>12.10</font></td>
<td><font color=#6F9915>11.37</font></td>
<td><font color=#6F9915>-0.724</font></td>
<td><font color=#6F9915>94.01%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q71</font></td>
<td><font color=#6F9915>2.41</font></td>
<td><font color=#6F9915>2.30</font></td>
<td><font color=#6F9915>-0.116</font></td>
<td><font color=#6F9915>95.19%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q72</font></td>
<td><font color=#6F9915>217.90</font></td>
<td><font color=#6F9915>212.84</font></td>
<td><font color=#6F9915>-5.057</font></td>
<td><font color=#6F9915>97.68%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q73</font></td>
<td><font color=#6F9915>2.46</font></td>
<td><font color=#6F9915>2.31</font></td>
<td><font color=#6F9915>-0.147</font></td>
<td><font color=#6F9915>94.01%</font></td>
</tr>
<tr>
<td><font color=#F92663>q74</font></td>
<td><font color=#F92663>24.09</font></td>
<td><font color=#F92663>24.25</font></td>
<td><font color=#F92663>0.156</font></td>
<td><font color=#F92663>100.65%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q75</font></td>
<td><font color=#6F9915>27.30</font></td>
<td><font color=#6F9915>26.53</font></td>
<td><font color=#6F9915>-0.769</font></td>
<td><font color=#6F9915>97.18%</font></td>
</tr>
<tr>
<td><font color=#F92663>q76</font></td>
<td><font color=#F92663>11.60</font></td>
<td><font color=#F92663>11.79</font></td>
<td><font color=#F92663>0.186</font></td>
<td><font color=#F92663>101.60%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q77</font></td>
<td><font color=#6F9915>2.44</font></td>
<td><font color=#6F9915>2.32</font></td>
<td><font color=#6F9915>-0.121</font></td>
<td><font color=#6F9915>95.04%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q78</font></td>
<td><font color=#6F9915>50.23</font></td>
<td><font color=#6F9915>49.67</font></td>
<td><font color=#6F9915>-0.558</font></td>
<td><font color=#6F9915>98.89%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q79</font></td>
<td><font color=#6F9915>4.08</font></td>
<td><font color=#6F9915>3.94</font></td>
<td><font color=#6F9915>-0.149</font></td>
<td><font color=#6F9915>96.36%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q80</font></td>
<td><font color=#6F9915>12.35</font></td>
<td><font color=#6F9915>12.33</font></td>
<td><font color=#6F9915>-0.023</font></td>
<td><font color=#6F9915>99.82%</font></td>
</tr>
<tr>
<td><font color=#F92663>q81</font></td>
<td><font color=#F92663>4.88</font></td>
<td><font color=#F92663>4.98</font></td>
<td><font color=#F92663>0.103</font></td>
<td><font color=#F92663>102.12%</font></td>
</tr>
<tr>
<td><font color=#F92663>q82</font></td>
<td><font color=#F92663>7.29</font></td>
<td><font color=#F92663>7.79</font></td>
<td><font color=#F92663>0.500</font></td>
<td><font color=#F92663>106.85%</font></td>
</tr>
<tr>
<td><font color=#F92663>q83</font></td>
<td><font color=#F92663>1.69</font></td>
<td><font color=#F92663>1.74</font></td>
<td><font color=#F92663>0.052</font></td>
<td><font color=#F92663>103.06%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q84</font></td>
<td><font color=#6F9915>2.86</font></td>
<td><font color=#6F9915>2.82</font></td>
<td><font color=#6F9915>-0.032</font></td>
<td><font color=#6F9915>98.89%</font></td>
</tr>
<tr>
<td><font color=#F92663>q85</font></td>
<td><font color=#F92663>7.12</font></td>
<td><font color=#F92663>8.36</font></td>
<td><font color=#F92663>1.241</font></td>
<td><font color=#F92663>117.45%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q86</font></td>
<td><font color=#6F9915>4.25</font></td>
<td><font color=#6F9915>4.09</font></td>
<td><font color=#6F9915>-0.161</font></td>
<td><font color=#6F9915>96.20%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q87</font></td>
<td><font color=#6F9915>15.68</font></td>
<td><font color=#6F9915>14.21</font></td>
<td><font color=#6F9915>-1.469</font></td>
<td><font color=#6F9915>90.63%</font></td>
</tr>
<tr>
<td><font color=#F92663>q88</font></td>
<td><font color=#F92663>20.60</font></td>
<td><font color=#F92663>21.80</font></td>
<td><font color=#F92663>1.203</font></td>
<td><font color=#F92663>105.84%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q89</font></td>
<td><font color=#6F9915>3.72</font></td>
<td><font color=#6F9915>3.51</font></td>
<td><font color=#6F9915>-0.205</font></td>
<td><font color=#6F9915>94.50%</font></td>
</tr>
<tr>
<td><font color=#F92663>q90</font></td>
<td><font color=#F92663>3.07</font></td>
<td><font color=#F92663>3.58</font></td>
<td><font color=#F92663>0.505</font></td>
<td><font color=#F92663>116.45%</font></td>
</tr>
<tr>
<td><font color=#F92663>q91</font></td>
<td><font color=#F92663>2.21</font></td>
<td><font color=#F92663>2.43</font></td>
<td><font color=#F92663>0.221</font></td>
<td><font color=#F92663>110.01%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q92</font></td>
<td><font color=#6F9915>1.36</font></td>
<td><font color=#6F9915>1.31</font></td>
<td><font color=#6F9915>-0.051</font></td>
<td><font color=#6F9915>96.23%</font></td>
</tr>
<tr>
<td><font color=#F92663>q93</font></td>
<td><font color=#F92663>39.22</font></td>
<td><font color=#F92663>39.93</font></td>
<td><font color=#F92663>0.707</font></td>
<td><font color=#F92663>101.80%</font></td>
</tr>
<tr>
<td><font color=#F92663>q94</font></td>
<td><font color=#F92663>24.32</font></td>
<td><font color=#F92663>24.47</font></td>
<td><font color=#F92663>0.150</font></td>
<td><font color=#F92663>100.62%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q9</font></td>
<td><font color=#6F9915>89.61</font></td>
<td><font color=#6F9915>85.89</font></td>
<td><font color=#6F9915>-3.720</font></td>
<td><font color=#6F9915>95.85%</font></td>
</tr>
<tr>
<td><font color=#6F9915>q5</font></td>
<td><font color=#6F9915>2.80</font></td>
<td><font color=#6F9915>2.65</font></td>
<td><font color=#6F9915>-0.153</font></td>
<td><font color=#6F9915>94.54%</font></td>
</tr>
<tr>
<td><font color=#F92663>q96</font></td>
<td><font color=#F92663>17.49</font></td>
<td><font color=#F92663>17.68</font></td>
<td><font color=#F92663>0.193</font></td>
<td><font color=#F92663>101.10%</font></td>
</tr>
<tr>
<td><font color=#F92663>q97</font></td>
<td><font color=#F92663>1.89</font></td>
<td><font color=#F92663>2.01</font></td>
<td><font color=#F92663>0.119</font></td>
<td><font color=#F92663>106.30%</font></td>
</tr>
<tr>
<td><font color=#F92663>q98</font></td>
<td><font color=#F92663>10.03</font></td>
<td><font color=#F92663>10.52</font></td>
<td><font color=#F92663>0.491</font></td>
<td><font color=#F92663>104.90%</font></td>
</tr>
<tr>
<td><font color=#F92663>q99</font></td>
<td><font color=#F92663>10.03</font></td>
<td><font color=#F92663>10.52</font></td>
<td><font color=#F92663>0.491</font></td>
<td><font color=#F92663>104.90%</font></td>
</tr>
<tr>
<td><font color=#6F9915>total</font></td>
<td><font color=#6F9915>2219.17</font></td>
<td><font color=#6F9915>2183.21</font></td>
<td><font color=#6F9915>-35.958</font></td>
<td><font color=#6F9915>98.38%</font></td>
</tr>
</tbody>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#discussion_r1728140430
##########
gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java:
##########
@@ -1255,9 +1266,8 @@ void setNull(int rowId) {
throw new UnsupportedOperationException();
}
- void setNotNull(int rowId) {
- throw new UnsupportedOperationException();
- }
+ // Arrow not need to setNotNull, set the valus is enough.
+ void setNotNull(int rowId) {}
Review Comment:
Can we route `putDecimal` calls to `writer.setDecimal`? I am not that familiar with the design here but looks to be a manner we should follow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#discussion_r1728180120
##########
gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java:
##########
@@ -1255,9 +1266,8 @@ void setNull(int rowId) {
throw new UnsupportedOperationException();
}
- void setNotNull(int rowId) {
- throw new UnsupportedOperationException();
- }
+ // Arrow not need to setNotNull, set the valus is enough.
+ void setNotNull(int rowId) {}
Review Comment:
writer does not have the `setDecimal` function, I copied most code from https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java#L429 but is different in putByteArray, because Arrow DecimalVector does not support put byte array range.
Put the all bytes is final function in Spark.
```
public abstract int putByteArray(int rowId, byte[] value, int offset, int count);
public final int putByteArray(int rowId, byte[] value) {
return putByteArray(rowId, value, 0, value.length);
}
```
I will add a test for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#discussion_r1726170128
##########
gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java:
##########
@@ -1255,9 +1266,8 @@ void setNull(int rowId) {
throw new UnsupportedOperationException();
}
- void setNotNull(int rowId) {
- throw new UnsupportedOperationException();
- }
+ // Arrow not need to setNotNull, set the valus is enough.
+ void setNotNull(int rowId) {}
Review Comment:
The class is abstract and the method should be overridable by impl classes.
See https://github.com/apache/incubator-gluten/blob/ed3d333c7c357e95f6736f378c972b714bdfb420/gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java#L1913 as example.
~~And abstract class `ArrowVectorWriter` doesn't likely have to be abstract. Could make it an interface as it almost doesn't override anything from the hierarchy. ~~ (needs more design)
##########
gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java:
##########
@@ -1255,9 +1266,8 @@ void setNull(int rowId) {
throw new UnsupportedOperationException();
}
- void setNotNull(int rowId) {
- throw new UnsupportedOperationException();
- }
+ // Arrow not need to setNotNull, set the valus is enough.
+ void setNotNull(int rowId) {}
Review Comment:
The class is abstract and the method should be overridable by impl classes.
See https://github.com/apache/incubator-gluten/blob/ed3d333c7c357e95f6736f378c972b714bdfb420/gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java#L1913 as example.
~~And abstract class `ArrowVectorWriter` doesn't likely have to be abstract. Could make it an interface as it almost doesn't override anything from the hierarchy.~~ (needs more design)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6961] [VL] [feat] ArrowWritableColumnVector support write to decimal [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #6962:
URL: https://github.com/apache/incubator-gluten/pull/6962#discussion_r1726178633
##########
gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java:
##########
@@ -1255,9 +1266,8 @@ void setNull(int rowId) {
throw new UnsupportedOperationException();
}
- void setNotNull(int rowId) {
- throw new UnsupportedOperationException();
- }
+ // Arrow not need to setNotNull, set the valus is enough.
+ void setNotNull(int rowId) {}
Review Comment:
Because it has member `ValueVector`, https://github.com/apache/incubator-gluten/blob/ed3d333c7c357e95f6736f378c972b714bdfb420/gluten-data/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java#L1244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org