You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "rkarthik29 (via GitHub)" <gi...@apache.org> on 2023/12/18 21:20:10 UTC
[I] pyarrow csv does not work with a cedilla delimiter [arrow]
rkarthik29 opened a new issue, #39287:
URL: https://github.com/apache/arrow/issues/39287
### Describe the bug, including details regarding any error messages, version, and platform.
we are trying to parse a CSV file that has a cedilla \x03 delimiter. When passing this to pyarrow.csv.read_csv, the parse complains that the delimiter \x03 is not a valid ascii. Is there a way around this issue ?
ile ~/miniconda3/envs/sandbox/lib/python3.11/site-packages/pyarrow/_csv.pyx:422, in pyarrow._csv.ParseOptions.__init__()
File ~/miniconda3/envs/sandbox/lib/python3.11/site-packages/pyarrow/_csv.pyx:445, in pyarrow._csv.ParseOptions.delimiter.__set__()
File ~/miniconda3/envs/sandbox/lib/python3.11/site-packages/pyarrow/_csv.pyx:46, in pyarrow._csv._single_char()
ValueError: Expecting an ASCII character
i get the above error if i use the character as delim "Ç"
but if i use \x03 , the data is read, but it is not parsed into columns.
i also tried "Ç".encode(ascii) and i get an error that the ascii code \xc7 is not valid. I tried chr(199). Nothing worked. Could anybody guide me in the right direction here. Dont want to use pandas
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] csv does not work with a cedilla delimiter [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche closed issue #39287: [Python] csv does not work with a cedilla delimiter
URL: https://github.com/apache/arrow/issues/39287
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] pyarrow csv does not work with a cedilla delimiter [arrow]
Posted by "rkarthik29 (via GitHub)" <gi...@apache.org>.
rkarthik29 commented on issue #39287:
URL: https://github.com/apache/arrow/issues/39287#issuecomment-1874386746
@AlenkaF It works with pandas if we use the unicode value \x03 for the cedilla as our seperator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] csv does not work with a cedilla delimiter [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #39287:
URL: https://github.com/apache/arrow/issues/39287#issuecomment-1881458769
There is currently no way around this: arrow only supports single ASCII characters as separator at the moment.
Related issues: https://github.com/apache/arrow/issues/32432, https://github.com/apache/arrow/issues/26411
Going to close this as duplicate of those other issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] pyarrow csv does not work with a cedilla delimiter [arrow]
Posted by "rkarthik29 (via GitHub)" <gi...@apache.org>.
rkarthik29 commented on issue #39287:
URL: https://github.com/apache/arrow/issues/39287#issuecomment-1875483991
reader=pd.read_csv(pfs.open("xyz.dat","rb")
,sep="\Ç",header=None,names=names,chunksize=100,encoding="utf-8",quotechar='"')
this works for pandas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] pyarrow csv does not work with a cedilla delimiter [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #39287:
URL: https://github.com/apache/arrow/issues/39287#issuecomment-1862311714
I think the issue here is that letter `C` with cedilla is not included in the main ASCII Table and I am not sure how to work around it.
I am not sure pandas will work here either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org