You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "n0r0shi (via GitHub)" <gi...@apache.org> on 2026/06/20 08:56:46 UTC

[PR] [GLUTEN-11570][VL] Wire spark.sql.binaryOutputStyle to Velox to_pretty_string [gluten]

n0r0shi opened a new pull request, #12324:
URL: https://github.com/apache/gluten/pull/12324

   ## What changes were proposed in this pull request?
   
   Spark 4.0+ introduced `spark.sql.binaryOutputStyle` to control how binary values are rendered. Velox's `to_pretty_string` previously only supported the default `HEX_DISCRETE` style. This change wires the Spark config through Gluten to Velox so all 5 styles (`HEX_DISCRETE`, `HEX`, `BASE64`, `UTF-8`, `BASIC`) are honored.
   
     - Add `kSparkBinaryOutputStyle = "spark.sql.binaryOutputStyle"` constant.
     - Forward it via `setIfExists` in `WholeStageResultIterator`, mapping to
       Velox's `SparkQueryConfig::kBinaryOutputStyle` (`binary_output_style`).
     - Expand `GlutenDataFrameSuite.getRows: binary` in spark40 and spark41 to
       cover all 5 styles, mirroring the upstream Spark `DataFrameSuite` test.
   
   ## Depends on
   
   Velox PR: https://github.com/facebookincubator/velox/pull/17884
   (adds `SparkQueryConfig::kBinaryOutputStyle` + 5 styles in `ToPrettyStringVarbinaryFunction`).
   
   Once that lands, this PR's behavior fully kicks in. With only this Gluten PR (and the Velox PR not yet merged), the existing `HEX_DISCRETE` behavior is preserved, but `HEX`/`BASE64`/`UTF-8`/`BASIC` will not take effect.
   
   ## How was this patch tested?
   
   `GlutenDataFrameSuite.getRows: binary` in spark40 and spark41; this test covers all 5 styles using `withSQLConf(SQLConf.BINARY_OUTPUT_STYLE.key -> ...)`, mirroring the upstream Spark `DataFrameSuite.getRows: binary` test.
   
     Fixes #11570
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [GLUTEN-11570][VL] Wire spark.sql.binaryOutputStyle to Velox to_pretty_string [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12324:
URL: https://github.com/apache/gluten/pull/12324#issuecomment-5466034681

   This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [GLUTEN-11570][VL] Wire spark.sql.binaryOutputStyle to Velox to_pretty_string [gluten]

Posted by "n0r0shi (via GitHub)" <gi...@apache.org>.
n0r0shi commented on PR #12324:
URL: https://github.com/apache/gluten/pull/12324#issuecomment-5965022993

   @FelixYBW Sorry for the late reply. Yes, I'm still working on it. I've updated the Velox PR (https://github.com/facebookincubator/velox/pull/17884) to address the review comments. I've also rebased this PR and fixed a bug: `spark.sql.binaryOutputStyle` wasn't passed to the native side. The native build will fail in CI until the Velox PR lands.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [GLUTEN-11570][VL] Wire spark.sql.binaryOutputStyle to Velox to_pretty_string [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12324:
URL: https://github.com/apache/gluten/pull/12324#issuecomment-5965013380

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [GLUTEN-11570][VL] Wire spark.sql.binaryOutputStyle to Velox to_pretty_string [gluten]

Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #12324:
URL: https://github.com/apache/gluten/pull/12324#issuecomment-6047898912

   @rui-mo can you take a look of https://github.com/facebookincubator/velox/pull/17884?
   
   @n0r0shi we may also move the spark sql functions to Gluten's cpp/velox/operators/functions before it's merged into velox. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [GLUTEN-11570][VL] Wire spark.sql.binaryOutputStyle to Velox to_pretty_string [gluten]

Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #12324:
URL: https://github.com/apache/gluten/pull/12324#issuecomment-5486410605

   @n0r0shi are you still working on it?
   
   cc @philo-he 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [GLUTEN-11570][VL] Wire spark.sql.binaryOutputStyle to Velox to_pretty_string [gluten]

Posted by "n0r0shi (via GitHub)" <gi...@apache.org>.
n0r0shi commented on PR #12324:
URL: https://github.com/apache/gluten/pull/12324#issuecomment-6138432873

   @FelixYBW Thanks for the suggestion! I've addressed all the review comments on https://github.com/facebookincubator/velox/pull/17884, so I'd prefer to land it in Velox first. If it stays stuck there for a while, I'll consider moving the implementation to the function overlay in Gluten.
   
   If this is urgent on your side, please let me know. I am happy to work on staging first.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [GLUTEN-11570][VL] Wire spark.sql.binaryOutputStyle to Velox to_pretty_string [gluten]

Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #12324:
URL: https://github.com/apache/gluten/pull/12324#issuecomment-4977732301

   @n0r0shi can you update https://github.com/facebookincubator/velox/pull/17884?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org