You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "L4XB (via GitHub)" <gi...@apache.org> on 2026/10/22 07:03:38 UTC
[PR] fix(csv): stop doubling backslashes in CSV exports [superset]
L4XB opened a new pull request, #45614:
URL: https://github.com/apache/superset/pull/45614
### SUMMARY
Every CSV export written by `df_to_escaped_csv` doubles each backslash in the data: chart and SQL Lab CSV downloads, the CSV result format, and CSV attachments in reports. A Windows path, a regular expression or an escaped JSON string comes out changed:
```python
df = pd.DataFrame({"path": ["C:\\Users\\ann"]})
df_to_escaped_csv(df, index=False)
# master: 'path\nC:\\\\Users\\\\ann\n' (the file contains C:\\Users\\ann)
# this PR: 'path\nC:\\Users\\ann\n' (the file contains C:\Users\ann)
```
`df_to_escaped_csv` passes `escapechar="\\"` to `DataFrame.to_csv`. #28755 added it as a workaround for pandas-dev/pandas#47871: on Python 3.10, the `csv` writer raised `need to escape, but no escapechar set` for a value holding a NUL character. With an escape character set, though, the writer also escapes that character itself in every field, so each `\` is written as `\\`. The test from #28755 pins that side effect: the escaped formula `'=cmd\|' /C calc'!A0` is expected as `'=cmd\\|' /C calc'!A0`.
Superset now requires Python 3.11 (`requires-python = ">=3.11"`), and from 3.11 on the writer writes a NUL without an escape character. I checked this on 3.11, 3.12 and 3.13; the NUL case in `test_df_to_escaped_csv` still passes. The escape character is therefore set only when the caller asks for `quoting=csv.QUOTE_NONE`, which has no other way to write a delimiter, quote or line break. A deployment whose `CSV_EXPORT` sets that keeps the current behaviour, and `escapechar` passed through `CSV_EXPORT` is respected.
### TESTING INSTRUCTIONS
- `pytest tests/unit_tests/utils/csv_tests.py`: 15 passed.
- New `test_df_to_escaped_csv_keeps_backslashes` checks the default export (backslashes kept) and `QUOTE_NONE` (escape character still applied). It fails on `master`.
- `test_df_to_escaped_csv` expects the escaped formula as `'=cmd\|' /C calc'!A0` again, the way it read before #28755. The formula escaping itself is unchanged.
- `tests/unit_tests/charts`, `commands/sql_lab`, `common`, `reports`, `result_set_test.py` and `dataframe_test.py` show no new failures. `test_df_to_records_preserves_finite_longdouble_and_nulls_nonfinite` and `test_get_data_json_preserves_browser_numeric_contract` fail the same way on `master` on this machine (arm64 macOS).
- `ruff check` / `ruff format` (0.9.7) are clean, and `mypy --check-untyped-defs` / `pylint` report nothing in `superset/utils/csv.py`.
To check by hand, run a SQL Lab query such as `SELECT 'C:\Users\ann' AS path` (escape the backslash as your database requires) and download the result as CSV. The file should contain `C:\Users\ann`.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(csv): stop doubling backslashes in CSV exports [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #45614:
URL: https://github.com/apache/superset/pull/45614#issuecomment-6334996203
## CodeAnt PR Risk: Medium Risk
- The PR needs attention before merging: with QUOTE_NONE and escapechar=None, fields containing delimiters or newlines make pandas raise an error.
**Assessed commit:** `b766bdb2732e`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(csv): stop doubling backslashes in CSV exports [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #45614:
URL: https://github.com/apache/superset/pull/45614#issuecomment-6335310382
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/45614?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `50.00000%` with `2 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.28%. Comparing base ([`b3471a1`](https://app.codecov.io/gh/apache/superset/commit/b3471a1369cf5a0527571b643839374854732dec?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b766bdb`](https://app.codecov.io/gh/apache/superset/commit/b766bdb2732ea05aec06f6231b3a519b208f4516?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/45614?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/utils/csv.py](https://app.codecov.io/gh/apache/superset/pull/45614?src=pr&el=tree&filepath=superset%2Futils%2Fcsv.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdXRpbHMvY3N2LnB5) | 50.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/superset/pull/45614?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (b3471a1) and HEAD (b766bdb). Click for more details.
>
> <details><summary>HEAD has 4 uploads less than BASE</summary>
>
>| Flag | BASE (b3471a1) | HEAD (b766bdb) |
>|------|------|------|
>|python|7|3|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #45614 +/- ##
===========================================
- Coverage 84.10% 65.28% -18.83%
===========================================
Files 3056 3056
Lines 201287 201290 +3
Branches 47066 47067 +1
===========================================
- Hits 169296 131407 -37889
- Misses 28752 66925 +38173
+ Partials 3239 2958 -281
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/45614/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ξ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/45614/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.89% <25.00%> (-0.01%)` | :arrow_down: |
| [mysql](https://app.codecov.io/gh/apache/superset/pull/45614/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [postgres](https://app.codecov.io/gh/apache/superset/pull/45614/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/45614/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `36.62% <25.00%> (-0.01%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/superset/pull/45614/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `52.73% <50.00%> (-34.62%)` | :arrow_down: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/45614/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `52.62% <50.00%> (-0.01%)` | :arrow_down: |
| [unit](https://app.codecov.io/gh/apache/superset/pull/45614/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/45614?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(csv): stop doubling backslashes in CSV exports [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #45614:
URL: https://github.com/apache/superset/pull/45614#discussion_r4345960156
##########
superset/utils/csv.py:
##########
@@ -115,7 +116,13 @@ def escape_values(v: Any) -> Union[str, Any]:
if isinstance(value, str):
df.at[label, name] = escape_value(value)
- return df.to_csv(escapechar="\\", **kwargs)
+ # An escape character makes the writer double every backslash in the data
+ # (C:\Users becomes C:\\Users). Only QUOTE_NONE, which cannot quote a
+ # delimiter or a line break, needs one; Python 3.10's writer also wanted one
+ # for a NUL character (#28755), which Python 3.11 writes without it.
+ if kwargs.get("quoting") == QUOTE_NONE:
+ kwargs.setdefault("escapechar", "\\")
Review Comment:
β
**CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved** as of `5a0fcb8`.
When `quoting` is `QUOTE_NONE` and `escapechar` is missing or explicitly `None`, the code now sets it to a backslash before calling `to_csv`.
<sub>If that's not right, unresolve this thread and CodeAnt will leave it open.</sub>
<!-- codeant-auto-resolve-reply -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(csv): stop doubling backslashes in CSV exports [superset]
Posted by "L4XB (via GitHub)" <gi...@apache.org>.
L4XB commented on code in PR #45614:
URL: https://github.com/apache/superset/pull/45614#discussion_r4345950619
##########
superset/utils/csv.py:
##########
@@ -115,7 +116,13 @@ def escape_values(v: Any) -> Union[str, Any]:
if isinstance(value, str):
df.at[label, name] = escape_value(value)
- return df.to_csv(escapechar="\\", **kwargs)
+ # An escape character makes the writer double every backslash in the data
+ # (C:\Users becomes C:\\Users). Only QUOTE_NONE, which cannot quote a
+ # delimiter or a line break, needs one; Python 3.10's writer also wanted one
+ # for a NUL character (#28755), which Python 3.11 writes without it.
+ if kwargs.get("quoting") == QUOTE_NONE:
+ kwargs.setdefault("escapechar", "\\")
Review Comment:
Fixed in 5a0fcb8da. Under `quoting=QUOTE_NONE`, an `escapechar` of `None` now counts as unset, and the default `\\` applies. Any other explicit `escapechar` from `CSV_EXPORT` is still used as given. `test_df_to_escaped_csv_keeps_backslashes` now checks that `escapechar=None` gives the same output as leaving it out. On the previous commit, that case raised `need to escape, but no escapechar set`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(csv): stop doubling backslashes in CSV exports [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #45614:
URL: https://github.com/apache/superset/pull/45614#discussion_r4345502733
##########
superset/utils/csv.py:
##########
@@ -115,7 +116,13 @@ def escape_values(v: Any) -> Union[str, Any]:
if isinstance(value, str):
df.at[label, name] = escape_value(value)
- return df.to_csv(escapechar="\\", **kwargs)
+ # An escape character makes the writer double every backslash in the data
+ # (C:\Users becomes C:\\Users). Only QUOTE_NONE, which cannot quote a
+ # delimiter or a line break, needs one; Python 3.10's writer also wanted one
+ # for a NUL character (#28755), which Python 3.11 writes without it.
+ if kwargs.get("quoting") == QUOTE_NONE:
+ kwargs.setdefault("escapechar", "\\")
Review Comment:
**Suggestion:** `setdefault` preserves an explicitly configured `escapechar=None`; with `quoting=QUOTE_NONE`, a field containing a delimiter or newline then makes pandas raise βneed to escape, but no escapechar set.β
**Assessment:** π `Major` Β· π `Occurrence: Rarely` Β· π·οΈ `Api mismatch`
[](https://docs.codeant.ai/cli/resolve-pr-comments-skill) [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=3e4ebfa46b16464fa2683f7324861aa1&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=3e4ebfa46b16464fa2683f7324861aa1&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/utils/csv.py
**Line:** 124:124
**Comment:**
*Api Mismatch: `setdefault` preserves an explicitly configured `escapechar=None`; with `quoting=QUOTE_NONE`, a field containing a delimiter or newline then makes pandas raise βneed to escape, but no escapechar set.β
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F45614&comment_hash=e32fc1f00aa5f535735a47827196bfeac8a7d7b76101196d3effa5d555dd9609&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F45614&comment_hash=e32fc1f00aa5f535735a47827196bfeac8a7d7b76101196d3effa5d555dd9609&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org