You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhztheplayer (via GitHub)" <gi...@apache.org> on 2025/07/14 03:17:38 UTC
[PR] [VL] Clarify the code objective of VeloxDataSourceJniWrapper#splitBlockByPartitionAndBucket [incubator-gluten]
zhztheplayer opened a new pull request, #10179:
URL: https://github.com/apache/incubator-gluten/pull/10179
Related to #3076
Currently `VeloxDataSourceJniWrapper#splitBlockByPartitionAndBucket` doesn't do exactly it's intended to do. It actually takes an array of data columns then directly select them out from the input columnar batch.
The patch fixes the code a bit by moving the partition keys -> data keys conversion down from Java code to native code so less confusing will be made by the code.
Not dynamic partition write is still not supported. The patch just does code cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Clarify the code objective of VeloxDataSourceJniWrapper#splitBlockByPartitionAndBucket [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10179:
URL: https://github.com/apache/incubator-gluten/pull/10179#issuecomment-3067698009
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICEGJ-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on Github Issues?
https://github.com/apache/incubator-gluten/issues
Then could you also rename ***commit message*** and ***pull request title*** in the following format?
[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}
See also:
* [Other pull requests](https://github.com/apache/incubator-gluten/pulls/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Clarify the code objective of VeloxDataSourceJniWrapper#splitBlockByPartitionAndBucket [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer merged PR #10179:
URL: https://github.com/apache/incubator-gluten/pull/10179
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Clarify the code objective of VeloxDataSourceJniWrapper#splitBlockByPartitionAndBucket [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #10179:
URL: https://github.com/apache/incubator-gluten/pull/10179#issuecomment-3067699708
@JkSelf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Clarify the code objective of VeloxDataSourceJniWrapper#splitBlockByPartitionAndBucket [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10179:
URL: https://github.com/apache/incubator-gluten/pull/10179#issuecomment-3067696932
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICEGJ-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on Github Issues?
https://github.com/apache/incubator-gluten/issues
Then could you also rename ***commit message*** and ***pull request title*** in the following format?
[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}
See also:
* [Other pull requests](https://github.com/apache/incubator-gluten/pulls/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Clarify the code objective of VeloxDataSourceJniWrapper#splitBlockByPartitionAndBucket [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #10179:
URL: https://github.com/apache/incubator-gluten/pull/10179#issuecomment-3067904714
Found unrelated CI failures:
```
2025-07-14T04:08:34.2384242Z - MapFromEntries
2025-07-14T04:08:34.6700540Z E20250714 04:08:34.669479 62099 Exceptions.h:66] Line: /work/ep/build-velox/build/velox_ep/velox/functions/sparksql/ArraySort.cpp:200, Function:operator(), Expression: not a scalar type! kind: UNKNOWN, Source: RUNTIME, ErrorCode: INVALID_STATE
2025-07-14T04:08:34.6709932Z 04:08:34.670 WARN org.apache.spark.sql.execution.GlutenFallbackReporter: Validation failed for plan: Project[QueryId=68681], due to:
2025-07-14T04:08:34.6710936Z - Native validation failed:
2025-07-14T04:08:34.6712238Z |- Validation failed due to exception caught at file:SubstraitToVeloxPlanValidator.cc line:1444 function:validate, thrown from file:ArraySort.cpp line:200 function:operator(), reason:not a scalar type! kind: UNKNOWN
2025-07-14T04:08:34.6713718Z
2025-07-14T04:08:34.7943561Z 'Sort Array' NOT use gluten
2025-07-14T04:08:34.7944062Z - Sort Array
2025-07-14T04:08:35.1864710Z 'Array contains' offload to gluten
2025-07-14T04:08:35.1865251Z - Array contains
2025-07-14T04:08:35.4933009Z 'ArraysOverlap' offload to gluten
2025-07-14T04:08:35.4934250Z - ArraysOverlap *** FAILED ***
2025-07-14T04:08:35.4935542Z Incorrect evaluation: arrays_overlap([1,2,3], null), actual: true, expected: null (GlutenTestsTrait.scala:281)
```
```
2025-07-14T04:01:39.0117921Z GlutenNullExpressionsSuite:
2025-07-14T04:01:40.0730295Z 'isnull and isnotnull' offload to gluten
2025-07-14T04:01:40.0730821Z - isnull and isnotnull
2025-07-14T04:01:40.0740666Z 'AssertNotNUll' offload to gluten
2025-07-14T04:01:40.0741333Z - AssertNotNUll
2025-07-14T04:01:40.2183862Z 'IsNaN' offload to gluten
2025-07-14T04:01:40.2184315Z - IsNaN
2025-07-14T04:01:40.3274452Z 'nanvl' offload to gluten
2025-07-14T04:01:40.3276137Z - nanvl
2025-07-14T04:01:41.6566270Z 'coalesce' offload to gluten
2025-07-14T04:01:41.6587108Z - coalesce *** FAILED ***
2025-07-14T04:01:41.6588062Z Incorrect evaluation: coalesce(null, [1,2,3], [1,2,3,null]), actual: WrappedArray(1, 2, 3, null), expected: List(1, 2, 3)
```
```
2025-07-14T04:08:40.8974647Z - Validation failed with exception from: ProjectExecTransformer, reason: Not supported to map spark function name to substrait function name: stringsplitsql(null, .), class name: StringSplitSQL.
2025-07-14T04:08:40.8975414Z
2025-07-14T04:08:40.9152562Z 04:08:40.914 WARN org.apache.spark.sql.execution.GlutenFallbackReporter: Validation failed for plan: Project[QueryId=68911], due to:
2025-07-14T04:08:40.9155164Z - Validation failed with exception from: ProjectExecTransformer, reason: Not supported to map spark function name to substrait function name: stringsplitsql(11.12.13, null), class name: StringSplitSQL.
2025-07-14T04:08:40.9156145Z
2025-07-14T04:08:40.9363797Z 'elementAt' NOT use gluten
2025-07-14T04:08:40.9364289Z - elementAt
2025-07-14T04:08:40.9374048Z 'correctly handles ElementAt nullability for arrays' offload to gluten
2025-07-14T04:08:40.9375005Z - correctly handles ElementAt nullability for arrays
2025-07-14T04:08:41.1122115Z 'Concat' offload to gluten
2025-07-14T04:08:41.1122583Z - Concat *** FAILED ***
2025-07-14T04:08:41.1123485Z Incorrect evaluation: concat([1,2,3], null), actual: WrappedArray(1, 2, 3, 1, 2, 3), expected: null (GlutenTestsTrait.scala:281)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Clarify the code objective of VeloxDataSourceJniWrapper#splitBlockByPartitionAndBucket [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10179:
URL: https://github.com/apache/incubator-gluten/pull/10179#issuecomment-3067697192
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICEGJ-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on Github Issues?
https://github.com/apache/incubator-gluten/issues
Then could you also rename ***commit message*** and ***pull request title*** in the following format?
[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}
See also:
* [Other pull requests](https://github.com/apache/incubator-gluten/pulls/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Clarify the code objective of VeloxDataSourceJniWrapper#splitBlockByPartitionAndBucket [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #10179:
URL: https://github.com/apache/incubator-gluten/pull/10179#issuecomment-3068689900
@zjuwangg Thanks for reviewing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org