You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "sayre1000 (via GitHub)" <gi...@apache.org> on 2024/03/26 14:57:50 UTC
[I] Maintaining Parquet Writer Ruby [arrow]
sayre1000 opened a new issue, #40798:
URL: https://github.com/apache/arrow/issues/40798
### Describe the usage question you have. Please include as many useful details as possible.
I am trying to use the `red-parquet` and `red-arrow` gems for an archival tool that will batch by days. I am using Ruby on Rails.
For some of our tables the data for an entire day is excessively large in memory (20GB) and could scale up in the future. This being the case I'd prefer to batch these archiving by 100,000 records.
However, we still want one unified parquet file for any given day. I had thought there would be a way to maintain a writer to one parquet file and write/stream it but I have been unable to figure out how to do so.
Is this something that can be done? Will I instead have to merge them after they are written by calling an external python script, then deleting the old ones?
### Component(s)
Parquet, Ruby
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Ruby][Parquet] Maintaining Parquet Writer Ruby [arrow]
Posted by "sayre1000 (via GitHub)" <gi...@apache.org>.
sayre1000 commented on issue #40798:
URL: https://github.com/apache/arrow/issues/40798#issuecomment-2032655212
@kou's solution with the GC.start seems to be doing a pretty good job, I can get up to 1,000,000 records at a time into one file on my local machine and I believe this sufficiently reduces the overhead for our purposes. Thank you so much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Ruby][Parquet] Maintaining Parquet Writer Ruby [arrow]
Posted by "route (via GitHub)" <gi...@apache.org>.
route commented on issue #40798:
URL: https://github.com/apache/arrow/issues/40798#issuecomment-2026844691
Hm seems to be growing but at the lower rate. Thank you!
I also faced this issue when running your example https://github.com/ruby-gnome/ruby-gnome/issues/1610
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Ruby][Parquet] Maintaining Parquet Writer Ruby [arrow]
Posted by "route (via GitHub)" <gi...@apache.org>.
route commented on issue #40798:
URL: https://github.com/apache/arrow/issues/40798#issuecomment-2026812004
@kou While writing in kinda batches the memory is ever increasing, meaning that we have to put the whole thing anyways in mem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Ruby][Parquet] Maintaining Parquet Writer Ruby [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #40798:
URL: https://github.com/apache/arrow/issues/40798#issuecomment-2024464767
How about using `Parquet::ArrowFileWriter#write_table`?
```ruby
require "parquet"
schema = Arrow::Schema.new(a: :int64,
b: :int64,
c: :int64,
d: :int64)
n_total_records = 10000000000
n_columns = schema.n_fields
n_rows = 4096
max_int64 = 2 ** 63 - 1
arrays = n_columns.times.collect do
Arrow::Int64Array.new(n_rows.times.collect {rand(max_int64)})
end
table = Arrow::Table.new(schema, arrays)
Parquet::ArrowFileWriter.open(schema, "output.parquet") do |writer|
while n_total_records > 0
writer.write_table(table, 1024 * 1024)
n_total_records -= table.n_rows
end
end
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Ruby][Parquet] Maintaining Parquet Writer Ruby [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #40798:
URL: https://github.com/apache/arrow/issues/40798#issuecomment-2026825444
You may call `GC.start` explicitly in your loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Ruby][Parquet] Maintaining Parquet Writer Ruby [arrow]
Posted by "sayre1000 (via GitHub)" <gi...@apache.org>.
sayre1000 closed issue #40798: [Ruby][Parquet] Maintaining Parquet Writer Ruby
URL: https://github.com/apache/arrow/issues/40798
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org