You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/10 21:20:24 UTC
[GitHub] [arrow] arjunsr1 opened a new issue, #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
arjunsr1 opened a new issue, #13847:
URL: https://github.com/apache/arrow/issues/13847
Hello @kou - I had previously written about an issue of timestamp [ms] turning into timestamp [s] when loading parquet file data into an arrow table. This time, I have a similar question, but this time, it has led to significant data alteration that is hard to recover.
There is a query we are hitting from a third party, and giving us a CSV file. The CSV file is loaded by calling Arrow::Table.load(csv_file_path), and the schema contains lots of data. Among that data, two of the fields are listed in table.schema.to_s as 'date64 [ms]' which is a timestamp that includes the day and gives a precision of milliseconds, and one of them is listed as uint16. However, when we are calling Arrow::Table.save(new_file_path.parquet), it seems that the function changes the `date64 [ms]` types to `date32 [day]`, and the `uint16` to `uint8`. As you can imagine, this is stripping timestamp data and turning it into just a representation of the date.
I was wondering if there is functionality in the `Arrow::Table.save` method that would allow to specify what type we want to save certain columns as. Do you know if parquet supports this `date64 [ms]` type at all? If you could help with a workaround that would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
Posted by GitBox <gi...@apache.org>.
kou commented on issue #13847:
URL: https://github.com/apache/arrow/issues/13847#issuecomment-1211527576
Could you provide a sample Ruby script that reproduces this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
Posted by GitBox <gi...@apache.org>.
drin commented on issue #13847:
URL: https://github.com/apache/arrow/issues/13847#issuecomment-1211293475
Hello! it looks like there's a relevant JIRA issue: [ARROW-9502](https://issues.apache.org/jira/browse/ARROW-9502).
So, an answer to 1 of your questions seems to be that parquet does not support date64. I am not sure what to recommend as a workaround as I haven't worked with the date types at all so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arjunsr1 commented on issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
Posted by GitBox <gi...@apache.org>.
arjunsr1 commented on issue #13847:
URL: https://github.com/apache/arrow/issues/13847#issuecomment-1211418093
@drin , I was unable to use the cast method to change the date64 column to type :timestamp or Arrow::TimeUnit::MILLI. It doesn't allow conversion to TimeUnit. However, I believe that the issue may actually begin in `csv_loader.rb`. I noticed that after loading from CSV, I have quantities within the rows that look like this:
2022-08-10T09:50:04-07:00
But the header for that row ends up becoming a date64 [ms] quantity, which I think is actually already losing the data. When I run this code: `new_data_table['created_time'].data.chunks.collect { |chunk| chunk.to_s }` I get the below as an output
`new_data_table['created_time'].data.chunks.collect { |chunk| chunk.to_s }`
["[\n 2022-08-10,\n 2022-08-10,\n 2022-08-10,\n 2022-08-10,\n 2022-08-10,\n 2022-08-10,\n 2022-08-10,\n 2022-08-10,\n 2022-08-10,\n 2022-08-10,\n 2022-08-10\n]"]
As you can see, the data chunk by chunk is already losing the time units.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arjunsr1 commented on issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
Posted by GitBox <gi...@apache.org>.
arjunsr1 commented on issue #13847:
URL: https://github.com/apache/arrow/issues/13847#issuecomment-1218406304
Thanks for all the help, @kou .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arjunsr1 commented on issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
Posted by GitBox <gi...@apache.org>.
arjunsr1 commented on issue #13847:
URL: https://github.com/apache/arrow/issues/13847#issuecomment-1217297409
Hello @kou - I had tested the changes locally and everything looked fine, but in a testing environment for one of our tables, the addition of column_types did not work in terms of changing the schema. I've attached the file and the code we're using. Any help in diagnosing the problem would be greatly appreciated.
```
ARROW_STRING_TYPE = :string
ARROW_UINT16_TYPE = :uint16
TIMESTAMP_MS = lambda { Arrow::TimestampDataType.new(Arrow::TimeUnit::MILLI) }
COLUMN_TYPES = {
group_name: ARROW_STRING_TYPE,
created_time: TIMESTAMP_MS.call,
completed_time: TIMESTAMP_MS.call,
source: ARROW_STRING_TYPE,
status: ARROW_STRING_TYPE,
runtime: :double,
is_single_query: ARROW_STRING_TYPE,
user_name: ARROW_STRING_TYPE,
user_email: ARROW_STRING_TYPE,
user_is_disabled: ARROW_STRING_TYPE,
is_looker_employee: ARROW_STRING_TYPE,
query_id: ARROW_UINT16_TYPE,
query_filters: ARROW_STRING_TYPE,
query_fields: ARROW_STRING_TYPE,
query_dynamic_fields: ARROW_STRING_TYPE,
instance_number: ARROW_UINT16_TYPE
}
table = Arrow::Table.load(path_to_file, column_types: COLUMN_TYPES)
# table.schema.to_s.split("\n") continues to show date64[ms] as well as other un-converted column types.
# Note that the code works on multiple other tables we had tried.
```
[instance_id8.csv](https://github.com/apache/arrow/files/9355369/instance_id8.csv)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arjunsr1 commented on issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
Posted by GitBox <gi...@apache.org>.
arjunsr1 commented on issue #13847:
URL: https://github.com/apache/arrow/issues/13847#issuecomment-1212293782
Hey @kou - I noticed that there is the capability to add column_types as an option passed into `Arrow::Table.load`. I'm trying to do this:
```
Arrow::Table.load(
tempfile.path, # this is the path of the CSV file
column_types: [%i[created_time time64], %i[completed_time time64]] # created_time and completed_time are the columns i want to change
)
```
However, I'm getting this error:
`wrong arguments: Arrow::TimestampDataType#initialize(): available signatures: (unit: interface(Arrow::TimeUnit(GArrowTimeUnit)))`
Can you give me any guidance on how to use the :column_types option?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arjunsr1 commented on issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
Posted by GitBox <gi...@apache.org>.
arjunsr1 commented on issue #13847:
URL: https://github.com/apache/arrow/issues/13847#issuecomment-1211297338
Oh, wow! Thanks for the speedy reply @drin.
As for the lack of parquet support, its unfortunate :( I'm going to try using the `cast` method that was added in red-arrow 9.0.0 to transform the date64 type to timestamp and see if that's something that's supported. Will check in back here after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
Posted by GitBox <gi...@apache.org>.
kou commented on issue #13847:
URL: https://github.com/apache/arrow/issues/13847#issuecomment-1217374281
Because some `query_id` values are over `uint16`. For example, the first record has `80956`. It's over `65535`.
`query_id: :uint32` will solve it.
FYI: `Arrow::Table.load` for CSV tries Apache Arrow C++'s CSV parser at first. If the try was failed, it uses Ruby's csv gem as fallback. The first try doesn't report details on error. We may need to add an option that forces to use Apache Arrow C++'s CSV parser.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arjunsr1 closed issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
Posted by GitBox <gi...@apache.org>.
arjunsr1 closed issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
URL: https://github.com/apache/arrow/issues/13847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arjunsr1 commented on issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
Posted by GitBox <gi...@apache.org>.
arjunsr1 commented on issue #13847:
URL: https://github.com/apache/arrow/issues/13847#issuecomment-1212635004
Update: I was able to fix it by writing this:
```
column_types: [
[:created_time, Arrow::TimestampDataType.new(Arrow::TimeUnit::MILLI)],
[:completed_time, Arrow::TimestampDataType.new(Arrow::TimeUnit::MILLI)],
%i[query_id uint16]
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
Posted by GitBox <gi...@apache.org>.
kou commented on issue #13847:
URL: https://github.com/apache/arrow/issues/13847#issuecomment-1212680571
Can we close this?
BTW, `Hash` is preferred than `Array`:
```ruby
column_types: {
created_time: Arrow::TimestampDataType.new(Arrow::TimeUnit::MILLI), # [:timestamp, :milli] will work
completed_time: Arrow::TimestampDataType.new(Arrow::TimeUnit::MILLI), # [:timestamp, :milli] will work
query_id: :uint16,
}
```
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
Posted by GitBox <gi...@apache.org>.
drin commented on issue #13847:
URL: https://github.com/apache/arrow/issues/13847#issuecomment-1211298934
no problem, good luck!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arjunsr1 closed issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
Posted by GitBox <gi...@apache.org>.
arjunsr1 closed issue #13847: Arrow::Table.save(path_to_file.parquet) changes data types within Schema.
URL: https://github.com/apache/arrow/issues/13847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org