You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/01 23:33:18 UTC
[GitHub] [spark] Kimahriman commented on pull request #34294: [SPARK-37019][SQL] Add codegen support to array transform
Kimahriman commented on pull request #34294:
URL: https://github.com/apache/spark/pull/34294#issuecomment-956889103
Already ran into an issue testing this out in a real use case where making it non-splittable caused a function to grow to big and revert back to interpreted, so I changed to just always use the atomic variable reference to read the value so it can be splittable. New sample codegen:
```
/* 039 */ final int project_numElements_0 = localtablescan_value_0.numElements();
/* 040 */
/* 041 */ ArrayData project_arrayData_0 = ArrayData.allocateArrayData(
/* 042 */ 4, project_numElements_0, " transform failed.");
/* 043 */
/* 044 */ for (int project_i_0 = 0; project_i_0 < project_numElements_0; project_i_0++) {
/* 045 */ if (localtablescan_value_0.isNullAt(project_i_0)) {
/* 046 */ ((java.util.concurrent.atomic.AtomicReference) references[2] /* x_229 */).set(null);
/* 047 */ } else {
/* 048 */ ((java.util.concurrent.atomic.AtomicReference) references[2] /* x_229 */).set(localtablescan_value_0.getInt(project_i_0));
/* 049 */ }
/* 050 */
/* 051 */ boolean project_isNull_3 = true;
/* 052 */ int project_value_3 = -1;
/* 053 */ Object project_tmpAtomic_0 = ((java.util.concurrent.atomic.AtomicReference) references[1] /* x_229 */).get();
/* 054 */ int project_value_4 = -1;
/* 055 */ boolean project_isNull_4 = project_tmpAtomic_0 == null;
/* 056 */ if (!project_isNull_4) {
/* 057 */ project_value_4 = (Integer)project_tmpAtomic_0;
/* 058 */ }
/* 059 */ if (!project_isNull_4) {
/* 060 */ project_isNull_3 = false; // resultCode could change nullability.
/* 061 */
/* 062 */ project_value_3 = project_value_4 + 1;
/* 063 */
/* 064 */ }
/* 065 */ boolean project_isNull_2 = project_isNull_3;
/* 066 */ int project_value_2 = project_value_3;
/* 067 */
/* 068 */ if (project_isNull_2) {
/* 069 */ project_arrayData_0.setNullAt(project_i_0);
/* 070 */ } else {
/* 071 */ project_arrayData_0.setInt(project_i_0, project_value_2);
/* 072 */ }
/* 073 */
/* 074 */ }
/* 075 */ project_value_0 = project_arrayData_0;
/* 076 */
/* 077 */ }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org