You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "SamAya21 (via GitHub)" <gi...@apache.org> on 2026/04/13 19:36:40 UTC
[PR] fix: address escape literal issue #21516 [datafusion]
SamAya21 opened a new pull request, #21599:
URL: https://github.com/apache/datafusion/pull/21599
Summary
This PR fixes Spark-compatible handling of escape sequences in SQL string literals #21516 .
The issue showed up in datafusion-spark string function behavior, but the root cause was not in soundex itself. The actual problem was that quoted SQL string literals were being converted into DataFusion literal expressions without unescaping sequences such as \t, \n, \\, \', and octal escapes.
As a result, literals like '\t hello' were treated as the two characters \ and t instead of a tab character followed by hello.
What changed
This change updates SQL value handling in datafusion/sql/src/expr/value.rs so that:
regular quoted string literals are unescaped before being converted to Expr::Literal
escaped string literals follow the same unescape path
common escape sequences are supported:
\0
\b
\n
\r
\t
\Z
\\
\'
\"
\%
\_
octal escapes of up to 3 digits are supported, such as \101
Why this belongs here
Although the failing behavior was observed in Spark string functions, the underlying bug was earlier in the SQL literal pipeline. parse_value(...) in value.rs was converting normal quoted strings directly with lit(s), preserving backslash escape text instead of producing the intended string value.
Fixing the issue at the value-conversion layer ensures all string functions receive the correct literal content.
Tests
Added unit tests covering:
tab, newline, and carriage return escapes
escaped quotes and backslashes
octal escapes
unknown escapes
trailing backslash behavior
Notes
While working on validation, I also ran into projection-name conflicts when selecting multiple literals that now resolve to the same final value like / and //. For SQL-level tests, this is avoided by aliasing projected literals, and updated test case with cargo insta review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: address escape literal issue #21516 [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #21599:
URL: https://github.com/apache/datafusion/pull/21599#issuecomment-4642813066
Hi @SamAya21 is there any update on this? Are you still interested in moving this forward? I can help you if you need any support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: address escape literal issue #21516 [datafusion]
Posted by "SamAya21 (via GitHub)" <gi...@apache.org>.
SamAya21 commented on PR #21599:
URL: https://github.com/apache/datafusion/pull/21599#issuecomment-4262473468
I ran cargo fit --all, I've been having issues running clippy, I had been getting errors for files unrelated to any changes I made (I think there's something wrong with my build environment, I didn't even have cmake). For the sqllogic tests command you gave is that a different test than what I ran in my PR request information or something else? Also, what directory would I run the command make -C testing/runner test-sqlite. In the directories I have ran it in so far it says testing/runner not found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: address escape literal issue #21516 [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #21599:
URL: https://github.com/apache/datafusion/pull/21599#issuecomment-4249816407
> Sorry about that, I hadn't seen I should've ran sqllogictests. How exactly would I run the linter locally or what webpage would I find that as I see that I had forgot to add the header to a test file I had created.
1. run cargo fmt: `cargo fmt --all`
2. run clippy: `cargo clippy --workspace --all-features --all-targets -- --deny=warnings`
3. run sqllogictest: `make -C testing/runner test-sqlite`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: address escape literal issue #21516 [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #21599:
URL: https://github.com/apache/datafusion/pull/21599#issuecomment-4242340042
Few points before the actual code review:
1. Always make sure you run the linter locally with all the tests passing
2. Follow the PT template
3. Please read the ai assisted pr guidelines https://datafusion.apache.org/contributor-guide/index.html#ai-assisted-contributions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: address escape literal issue #21516 [datafusion]
Posted by "SamAya21 (via GitHub)" <gi...@apache.org>.
SamAya21 commented on PR #21599:
URL: https://github.com/apache/datafusion/pull/21599#issuecomment-4272498349
<img width="1018" height="60" alt="Screenshot 2026-04-17 at 10 55 08 PM" src="https://github.com/user-attachments/assets/dc7d9fe0-bdf3-453f-83da-e0556ea836e6" />
I ran sqlite, but this progress bar remained after it completing. Did all the files run or did it get stuck in this test file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: address escape literal issue #21516 [datafusion]
Posted by "SamAya21 (via GitHub)" <gi...@apache.org>.
SamAya21 commented on PR #21599:
URL: https://github.com/apache/datafusion/pull/21599#issuecomment-4247156169
Sorry about that, I hadn't seen I should've ran sqllogictests. How exactly would I run the linter locally or what webpage would I find that as I see that I had forgot to add the header to a test file I had created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: address escape literal issue #21516 [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21599:
URL: https://github.com/apache/datafusion/pull/21599#issuecomment-5350266967
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: address escape literal issue #21516 [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #21599:
URL: https://github.com/apache/datafusion/pull/21599#issuecomment-4265373578
https://datafusion.apache.org/contributor-guide/testing.html#testing-quick-start
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: address escape literal issue #21516 [datafusion]
Posted by "SamAya21 (via GitHub)" <gi...@apache.org>.
SamAya21 commented on PR #21599:
URL: https://github.com/apache/datafusion/pull/21599#issuecomment-4703508208
@kumarUjjawal I think I am done working on it. Sorry for the delay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: address escape literal issue #21516 [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #21599:
URL: https://github.com/apache/datafusion/pull/21599#issuecomment-4704249693
@SamAya21 Can you resolve the conflict and run `./dev/update_config_docs.sh` and `cargo fmt --all -- --check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: address escape literal issue #21516 [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #21599: fix: address escape literal issue #21516
URL: https://github.com/apache/datafusion/pull/21599
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org