You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/05/12 05:45:13 UTC
[GitHub] [arrow] micomahesh1982 commented on issue #13125: parquet conversion failed,Bool column has NA values in column boolean__v
micomahesh1982 commented on issue #13125:
URL: https://github.com/apache/arrow/issues/13125#issuecomment-1124554021
technically,
csv_file = StringIO("""int__v|Decimal__v|Float__v|Boolean__v|String__v|Null__v|Date__v|Timestamp__v
1|43.4|11.02|True|'456'|12|2021-03-02|2019-08-07 10:11:12
2|101.4|11.128|False|'456'||2020-09-09|2019-05-04 11:12:13
3|43.4|13.02|True|'456'||2022-03-14|2012-08-07 10:15:12
4|202.4|14.128|False|'456'||2020-03-15|2017-09-04 11:17:13
5|43.4|11.02|True|'456'||2021-03-02|2019-08-07 10:11:12
6|101.4|11.128|False|'456'||2020-09-09|2019-05-04 11:12:13
7|43.4|13.02|True|'456'||2022-03-14|2012-08-07 10:15:12
8|202.4|14.128|False|'456'||2020-03-15|2017-09-04 11:17:13
9|43.4|11.02|True|'456'||2021-03-02|2019-08-07 10:11:12
10|101.4|11.128|False|'456'||2020-09-09|2019-05-04 11:12:13
11|43.4|13.02|True|'456'||2022-03-14|2012-08-07 10:15:12
12|202.4|14.128|False|'456'||2020-03-15|2017-09-04 11:17:13
13|101.4|11.128|False|'456'||2020-09-09|2019-05-04 11:12:13
14|43.4|13.02|True|'456'||2022-03-14|2012-08-07 10:15:12
15|202.4|14.128|False|'456'||2020-03-15|2017-09-04 11:17:13
16|43.4|11.02|True|'456'||2021-03-02|2019-08-07 10:11:12
17|101.4|11.128|False|'456'||2020-09-09|2019-05-04 11:12:13
18|43.4|13.02|True|'456'||2022-03-14|2012-08-07 10:15:12
19|202.4|14.128|False|'456'||2020-03-15|2017-09-04 11:17:13
4|||||||
""")
params = { 'filepath_or_buffer': csv_file, 'chunksize': 10, 'encoding': 'UTF-8', 'sep': '|', 'low_memory': True, 'engine': 'python', 'skip_blank_lines': True }
Next, loop through chunk of data and convert csv files into parquet and write into s3.
you would be noticing 4th column is 'boolean' datatype and last row has 'NA' which means null data. That's where getting "parquet conversion failed,Bool column has NA values in column boolean__v"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org