You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/06/10 13:08:21 UTC
[GitHub] [arrow] pitrou opened a new pull request #10503: ARROW-10115: [C++] Add CSV option to treat quoted strings as always non-null
pitrou opened a new pull request #10503:
URL: https://github.com/apache/arrow/pull/10503
The option is only applicable to string and binary columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #10503: ARROW-10115: [C++] Add CSV option to treat quoted strings as always non-null
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #10503:
URL: https://github.com/apache/arrow/pull/10503#issuecomment-861564976
Pandas has loads of options, but doesn't cover this specific use case ;) (based on my [comment](https://issues.apache.org/jira/browse/ARROW-10115?focusedCommentId=17250151&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17250151) on the JIRA)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10503: ARROW-10115: [C++] Add CSV option to treat quoted strings as always non-null
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10503:
URL: https://github.com/apache/arrow/pull/10503#issuecomment-858622594
https://issues.apache.org/jira/browse/ARROW-10115
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mskrzypkows commented on pull request #10503: ARROW-10115: [C++] Add CSV option to treat quoted strings as always non-null
Posted by GitBox <gi...@apache.org>.
mskrzypkows commented on pull request #10503:
URL: https://github.com/apache/arrow/pull/10503#issuecomment-862172474
> Looks good, this indeed enables the use case from the JIRA nicely (I am only wondering if the default shouldn't be `quoted_strings_can_be_null=False`, although that's not backwards compatible)
I agree :-) It would be more intuitive for me to have it `false` by default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10503: ARROW-10115: [C++] Add CSV option to treat quoted strings as always non-null
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10503:
URL: https://github.com/apache/arrow/pull/10503#issuecomment-862191654
It seems Pandas detects nulls in quoted strings. @jorisvandenbossche Do you think we should do differently by default?
```python
>>> s = '''a,b\n1.0,"foo"\n"N/A","N/A"'''
>>> df = pd.read_csv(io.BytesIO(s.encode()))
>>> df
a b
0 1.0 foo
1 NaN NaN
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10503: ARROW-10115: [C++] Add CSV option to treat quoted strings as always non-null
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10503:
URL: https://github.com/apache/arrow/pull/10503#discussion_r651780106
##########
File path: cpp/src/arrow/csv/options.h
##########
@@ -74,11 +74,18 @@ struct ARROW_EXPORT ConvertOptions {
std::vector<std::string> true_values;
/// Recognized spellings for boolean false values
std::vector<std::string> false_values;
+
/// Whether string / binary columns can have null values.
///
/// If true, then strings in "null_values" are considered null for string columns.
/// If false, then all strings are valid string values.
bool strings_can_be_null = false;
+ /// Whether string / binary columns can have quotednull values.
Review comment:
```suggestion
/// Whether string / binary columns can have quoted null values.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10503: ARROW-10115: [C++] Add CSV option to treat quoted strings as always non-null
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10503:
URL: https://github.com/apache/arrow/pull/10503#issuecomment-861560048
> I am only wondering if the default shouldn't be quoted_strings_can_be_null=False, although that's not backwards compatible
Feel free to open a JIRA for that. What's the Pandas default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org