You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Micah Kornfield <em...@gmail.com> on 2020/07/10 04:46:49 UTC
Re: Writing very large rowgroups to Apache Parquet
+parquet-dev as this seems more concerned with the non-arrow pieces of
parquet
Hi Roman,
Answers inline.
One way to solve that problem would be to use memory mapped files instead
> of plain memory buffers. That way, the number of required memory can be
> limited by the number of columns times the os-pagesize, which would be
> independent of the rowgroup-size. Consequently, large rowgroupsizes pose no
> problem with respect to RAM consumption.
I was under the impression that modern allocator (i.e. jemalloc) already
mmap for large allocations. How would this approach be different from the
way allocators use it? Have you prototyped this approach to see if it
allows for better scalability?
> After a quick look at how the buffers are managed inside arrow (allocated
> from a default memory pool), I have the impression that an implementation
> of this idea could be a rather huge change. I still wanted to know whether
> that is something you could see being integrated or whether that is out of
> scope of arrow.
A huge change probably isn't a great idea unless we've validated the
approach along with alternatives. Is there currently code that doesn't
make use of the MemoryPool [1] provided by WriterProperties? If so we
should probably fix it. Otherwise, is there a reason that you can't
substitute a customized memory pool on WriterProperties?
Thanks,
Micah
[1]
https://github.com/apache/arrow/blob/5602c459eb8773b6be8059b1b118175e9f16b7a3/cpp/src/parquet/properties.h#L447
On Thu, Jul 9, 2020 at 8:35 AM Roman Karlstetter <
roman.karlstetter@gmail.com> wrote:
> Hi everyone,
>
> since some time now, parquet::ParquetFileWriter has the option to create
> buffered rowgroups with AppendBufferedRowGroup(), which basically gives you
> the possibility to write to columns in any order you like (in contrast to
> the former only possible way of writing one column after the other). This
> is cool since it avoids the caller from having to create an in memory
> columnar representation of its data.
>
> However, when data size is huge compared to the available system memory
> (due to wide schema or a large rowgroupsize), this is problematic, as the
> buffers allocated internally can take up a large portion of RAM of the
> machine the conversion is running on.
>
> One way to solve that problem would be to use memory mapped files instead
> of plain memory buffers. That way, the number of required memory can be
> limited by the number of columns times the os-pagesize, which would be
> independent of the rowgroup-size. Consequently, large rowgroupsizes pose no
> problem with respect to RAM consumption.
>
> I wonder what you generally think about the idea of integrating an
> AppendFileBufferedRowGroup() (or similar name) possibility which gives the
> user the option to have the internal buffers be memory mapped files.
>
> After a quick look at how the buffers are managed inside arrow (allocated
> from a default memory pool), I have the impression that an implementation
> of this idea could be a rather huge change. I still wanted to know whether
> that is something you could see being integrated or whether that is out of
> scope of arrow.
>
> Thanks in advance and kind regards,
> Roman
>
Re: Writing very large rowgroups to Apache Parquet
Posted by Micah Kornfield <em...@gmail.com>.
I did a quick search in Parquet-MR and found at least one place where
different files are explicitly forbidden [1]. I don't know if this blocks
all reading or is a specific case (I'm not sure if writing is allowed for
multiple columns).
Like I said, it makes sense, but is potentially a big change code wise (and
it appears at least in some cases would require updates across at least the
C++ and parquet-mr implementations).
[1]
https://github.com/apache/parquet-mr/blob/2589cc821d2d470be1e79b86f511eb1f5fee4e5c/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java#L1242
On Fri, Jul 17, 2020 at 9:48 AM Jacques Nadeau <ja...@apache.org> wrote:
> I believe the formal Parquet standard already allows a file per column. At
> least I remember it being discussed when the spec was first implemented. If
> you look at the thrift spec it actually allows for this:
>
>
> https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L771
>
> That being said, I'm not sure which readers support this read pattern. If
> it is part of the spec, doing it as a Parquet writing mode makes sense to
> me.
>
> On Mon, Jul 13, 2020 at 11:08 PM Roman Karlstetter <
> roman.karlstetter@gmail.com> wrote:
>
> > > I'd suggest a new write pattern. Write the columns page at a time to
> > separate files then use a second process to concatenate the columns and
> > append the footer. Odds are you would do better than os swapping and take
> > memory requirements down to
> > page size times field count.
> >
> > This is exactly what a student of us implemented pretty successfully:
> > writing to one file per column (non-parquet, binary, memory-mapped). And
> > once enough data is put into those "cache/buffer-files", the data is
> > flushed to a parquet rowgroup.
> >
> > My question targeted the integration of these ideas into the arrow
> parquet
> > writer. I wanted to know whether it makes sense to integrate these ideas
> or
> > whether it's better to keep that functionality outside of arrow/parquet.
> > Having it inside would have the benefit of reduced storage space because
> of
> > encoding/compression and thus smaller overhead in the final copy phase
> > (less data to copy and data already encoded/compressed). But on the other
> > hand, having one memory mapped file per column is not something that
> seems
> > to fit well with the current design of arrow.
> >
> > Thanks for the feedback,
> > Roman
> >
> > Am So., 12. Juli 2020 um 03:05 Uhr schrieb Micah Kornfield <
> > emkornfield@gmail.com>:
> >
> >> This is an interesting idea. For s3 multipart uploads one might run
> into
> >> limitations pretty quickly (only 10k parts appear to be supported. all
> but
> >> the last are expected to be at least 5mb if I read their docs correctly
> >> [1])
> >>
> >> [1] https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html
> >>
> >>
> >> On Saturday, July 11, 2020, Jacques Nadeau <ja...@apache.org> wrote:
> >>
> >> > I'd suggest a new write pattern. Write the columns page at a time to
> >> > separate files then use a second process to concatenate the columns
> and
> >> > append the footer. Odds are you would do better than os swapping and
> >> take
> >> > memory requirements down to page size times field count.
> >> >
> >> > In s3 I believe you could do this via a multipart upload and entirely
> >> skip
> >> > the second step. I don't know of any implementations that actually do
> >> this
> >> > yet.
> >> >
> >> > On Thu, Jul 9, 2020, 11:58 PM Roman Karlstetter <
> >> > roman.karlstetter@gmail.com> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I wasn't aware of the fact that jemalloc mmap automatically for
> larger
> >> >> allocations. And I didn't yet test this.
> >> >>
> >> >> The approach could be different in that we would know which parts of
> >> the
> >> >> buffers are going to be used next (the buffers are appendonly) and
> >> which
> >> >> parts won't be needed until actually flushing the rowgroup (and when
> >> >> flushing, we also know the order). But I'm not sure whether that
> >> knowledge
> >> >> helps a lot in a) saving memory compared to a generic allocator or b)
> >> >> improving performance. In addition to that, communicating this
> >> knowledge
> >> >> to
> >> >> the implementation will also be tricky for the general case, I guess.
> >> >>
> >> >> Regarding setting the allocator to another memory pool: I was unsure
> >> >> whether the memory pool is used for further allocations where the
> >> default
> >> >> memory pool would be more appropriate. If not, then setting the
> memory
> >> >> pool
> >> >> in the writer properties should actually work well.
> >> >>
> >> >> Maybe I should just play a bit with the different memory pool options
> >> and
> >> >> see how they behave. It makes more sense to discuss further ideas
> once
> >> I
> >> >> have some performance numbers.
> >> >>
> >> >> Thanks,
> >> >> Roman
> >> >>
> >> >>
> >> >> Am Fr., 10. Juli 2020 um 06:47 Uhr schrieb Micah Kornfield <
> >> >> emkornfield@gmail.com>:
> >> >>
> >> >> > +parquet-dev as this seems more concerned with the non-arrow pieces
> >> of
> >> >> > parquet
> >> >> >
> >> >> > Hi Roman,
> >> >> > Answers inline.
> >> >> >
> >> >> > One way to solve that problem would be to use memory mapped files
> >> >> instead
> >> >> > > of plain memory buffers. That way, the number of required memory
> >> can
> >> >> be
> >> >> > > limited by the number of columns times the os-pagesize, which
> >> would be
> >> >> > > independent of the rowgroup-size. Consequently, large
> rowgroupsizes
> >> >> pose
> >> >> > no
> >> >> > > problem with respect to RAM consumption.
> >> >> >
> >> >> > I was under the impression that modern allocator (i.e. jemalloc)
> >> already
> >> >> > mmap for large allocations. How would this approach be different
> >> from
> >> >> the
> >> >> > way allocators use it? Have you prototyped this approach to see if
> >> it
> >> >> > allows for better scalability?
> >> >> >
> >> >> >
> >> >> > > After a quick look at how the buffers are managed inside arrow
> >> >> (allocated
> >> >> > > from a default memory pool), I have the impression that an
> >> >> implementation
> >> >> > > of this idea could be a rather huge change. I still wanted to
> know
> >> >> > whether
> >> >> > > that is something you could see being integrated or whether that
> is
> >> >> out
> >> >> > of
> >> >> > > scope of arrow.
> >> >> >
> >> >> >
> >> >> > A huge change probably isn't a great idea unless we've validated
> the
> >> >> > approach along with alternatives. Is there currently code that
> >> doesn't
> >> >> > make use of the MemoryPool [1] provided by WriterProperties? If so
> we
> >> >> > should probably fix it. Otherwise, is there a reason that you
> can't
> >> >> > substitute a customized memory pool on WriterProperties?
> >> >> >
> >> >> > Thanks,
> >> >> > Micah
> >> >> >
> >> >> > [1]
> >> >> >
> >> >> >
> https://github.com/apache/arrow/blob/5602c459eb8773b6be8059b1b11817
> >> >> 5e9f16b7a3/cpp/src/parquet/properties.h#L447
> >> >> >
> >> >> > On Thu, Jul 9, 2020 at 8:35 AM Roman Karlstetter <
> >> >> > roman.karlstetter@gmail.com> wrote:
> >> >> >
> >> >> > > Hi everyone,
> >> >> > >
> >> >> > > since some time now, parquet::ParquetFileWriter has the option to
> >> >> create
> >> >> > > buffered rowgroups with AppendBufferedRowGroup(), which basically
> >> >> gives
> >> >> > you
> >> >> > > the possibility to write to columns in any order you like (in
> >> >> contrast to
> >> >> > > the former only possible way of writing one column after the
> >> other).
> >> >> This
> >> >> > > is cool since it avoids the caller from having to create an in
> >> memory
> >> >> > > columnar representation of its data.
> >> >> > >
> >> >> > > However, when data size is huge compared to the available system
> >> >> memory
> >> >> > > (due to wide schema or a large rowgroupsize), this is
> problematic,
> >> as
> >> >> the
> >> >> > > buffers allocated internally can take up a large portion of RAM
> of
> >> the
> >> >> > > machine the conversion is running on.
> >> >> > >
> >> >> > > One way to solve that problem would be to use memory mapped files
> >> >> instead
> >> >> > > of plain memory buffers. That way, the number of required memory
> >> can
> >> >> be
> >> >> > > limited by the number of columns times the os-pagesize, which
> >> would be
> >> >> > > independent of the rowgroup-size. Consequently, large
> rowgroupsizes
> >> >> pose
> >> >> > no
> >> >> > > problem with respect to RAM consumption.
> >> >> > >
> >> >> > > I wonder what you generally think about the idea of integrating
> an
> >> >> > > AppendFileBufferedRowGroup() (or similar name) possibility which
> >> gives
> >> >> > the
> >> >> > > user the option to have the internal buffers be memory mapped
> >> files.
> >> >> > >
> >> >> > > After a quick look at how the buffers are managed inside arrow
> >> >> (allocated
> >> >> > > from a default memory pool), I have the impression that an
> >> >> implementation
> >> >> > > of this idea could be a rather huge change. I still wanted to
> know
> >> >> > whether
> >> >> > > that is something you could see being integrated or whether that
> is
> >> >> out
> >> >> > of
> >> >> > > scope of arrow.
> >> >> > >
> >> >> > > Thanks in advance and kind regards,
> >> >> > > Roman
> >> >> > >
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: Writing very large rowgroups to Apache Parquet
Posted by Micah Kornfield <em...@gmail.com>.
I did a quick search in Parquet-MR and found at least one place where
different files are explicitly forbidden [1]. I don't know if this blocks
all reading or is a specific case (I'm not sure if writing is allowed for
multiple columns).
Like I said, it makes sense, but is potentially a big change code wise (and
it appears at least in some cases would require updates across at least the
C++ and parquet-mr implementations).
[1]
https://github.com/apache/parquet-mr/blob/2589cc821d2d470be1e79b86f511eb1f5fee4e5c/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java#L1242
On Fri, Jul 17, 2020 at 9:48 AM Jacques Nadeau <ja...@apache.org> wrote:
> I believe the formal Parquet standard already allows a file per column. At
> least I remember it being discussed when the spec was first implemented. If
> you look at the thrift spec it actually allows for this:
>
>
> https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L771
>
> That being said, I'm not sure which readers support this read pattern. If
> it is part of the spec, doing it as a Parquet writing mode makes sense to
> me.
>
> On Mon, Jul 13, 2020 at 11:08 PM Roman Karlstetter <
> roman.karlstetter@gmail.com> wrote:
>
> > > I'd suggest a new write pattern. Write the columns page at a time to
> > separate files then use a second process to concatenate the columns and
> > append the footer. Odds are you would do better than os swapping and take
> > memory requirements down to
> > page size times field count.
> >
> > This is exactly what a student of us implemented pretty successfully:
> > writing to one file per column (non-parquet, binary, memory-mapped). And
> > once enough data is put into those "cache/buffer-files", the data is
> > flushed to a parquet rowgroup.
> >
> > My question targeted the integration of these ideas into the arrow
> parquet
> > writer. I wanted to know whether it makes sense to integrate these ideas
> or
> > whether it's better to keep that functionality outside of arrow/parquet.
> > Having it inside would have the benefit of reduced storage space because
> of
> > encoding/compression and thus smaller overhead in the final copy phase
> > (less data to copy and data already encoded/compressed). But on the other
> > hand, having one memory mapped file per column is not something that
> seems
> > to fit well with the current design of arrow.
> >
> > Thanks for the feedback,
> > Roman
> >
> > Am So., 12. Juli 2020 um 03:05 Uhr schrieb Micah Kornfield <
> > emkornfield@gmail.com>:
> >
> >> This is an interesting idea. For s3 multipart uploads one might run
> into
> >> limitations pretty quickly (only 10k parts appear to be supported. all
> but
> >> the last are expected to be at least 5mb if I read their docs correctly
> >> [1])
> >>
> >> [1] https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html
> >>
> >>
> >> On Saturday, July 11, 2020, Jacques Nadeau <ja...@apache.org> wrote:
> >>
> >> > I'd suggest a new write pattern. Write the columns page at a time to
> >> > separate files then use a second process to concatenate the columns
> and
> >> > append the footer. Odds are you would do better than os swapping and
> >> take
> >> > memory requirements down to page size times field count.
> >> >
> >> > In s3 I believe you could do this via a multipart upload and entirely
> >> skip
> >> > the second step. I don't know of any implementations that actually do
> >> this
> >> > yet.
> >> >
> >> > On Thu, Jul 9, 2020, 11:58 PM Roman Karlstetter <
> >> > roman.karlstetter@gmail.com> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I wasn't aware of the fact that jemalloc mmap automatically for
> larger
> >> >> allocations. And I didn't yet test this.
> >> >>
> >> >> The approach could be different in that we would know which parts of
> >> the
> >> >> buffers are going to be used next (the buffers are appendonly) and
> >> which
> >> >> parts won't be needed until actually flushing the rowgroup (and when
> >> >> flushing, we also know the order). But I'm not sure whether that
> >> knowledge
> >> >> helps a lot in a) saving memory compared to a generic allocator or b)
> >> >> improving performance. In addition to that, communicating this
> >> knowledge
> >> >> to
> >> >> the implementation will also be tricky for the general case, I guess.
> >> >>
> >> >> Regarding setting the allocator to another memory pool: I was unsure
> >> >> whether the memory pool is used for further allocations where the
> >> default
> >> >> memory pool would be more appropriate. If not, then setting the
> memory
> >> >> pool
> >> >> in the writer properties should actually work well.
> >> >>
> >> >> Maybe I should just play a bit with the different memory pool options
> >> and
> >> >> see how they behave. It makes more sense to discuss further ideas
> once
> >> I
> >> >> have some performance numbers.
> >> >>
> >> >> Thanks,
> >> >> Roman
> >> >>
> >> >>
> >> >> Am Fr., 10. Juli 2020 um 06:47 Uhr schrieb Micah Kornfield <
> >> >> emkornfield@gmail.com>:
> >> >>
> >> >> > +parquet-dev as this seems more concerned with the non-arrow pieces
> >> of
> >> >> > parquet
> >> >> >
> >> >> > Hi Roman,
> >> >> > Answers inline.
> >> >> >
> >> >> > One way to solve that problem would be to use memory mapped files
> >> >> instead
> >> >> > > of plain memory buffers. That way, the number of required memory
> >> can
> >> >> be
> >> >> > > limited by the number of columns times the os-pagesize, which
> >> would be
> >> >> > > independent of the rowgroup-size. Consequently, large
> rowgroupsizes
> >> >> pose
> >> >> > no
> >> >> > > problem with respect to RAM consumption.
> >> >> >
> >> >> > I was under the impression that modern allocator (i.e. jemalloc)
> >> already
> >> >> > mmap for large allocations. How would this approach be different
> >> from
> >> >> the
> >> >> > way allocators use it? Have you prototyped this approach to see if
> >> it
> >> >> > allows for better scalability?
> >> >> >
> >> >> >
> >> >> > > After a quick look at how the buffers are managed inside arrow
> >> >> (allocated
> >> >> > > from a default memory pool), I have the impression that an
> >> >> implementation
> >> >> > > of this idea could be a rather huge change. I still wanted to
> know
> >> >> > whether
> >> >> > > that is something you could see being integrated or whether that
> is
> >> >> out
> >> >> > of
> >> >> > > scope of arrow.
> >> >> >
> >> >> >
> >> >> > A huge change probably isn't a great idea unless we've validated
> the
> >> >> > approach along with alternatives. Is there currently code that
> >> doesn't
> >> >> > make use of the MemoryPool [1] provided by WriterProperties? If so
> we
> >> >> > should probably fix it. Otherwise, is there a reason that you
> can't
> >> >> > substitute a customized memory pool on WriterProperties?
> >> >> >
> >> >> > Thanks,
> >> >> > Micah
> >> >> >
> >> >> > [1]
> >> >> >
> >> >> >
> https://github.com/apache/arrow/blob/5602c459eb8773b6be8059b1b11817
> >> >> 5e9f16b7a3/cpp/src/parquet/properties.h#L447
> >> >> >
> >> >> > On Thu, Jul 9, 2020 at 8:35 AM Roman Karlstetter <
> >> >> > roman.karlstetter@gmail.com> wrote:
> >> >> >
> >> >> > > Hi everyone,
> >> >> > >
> >> >> > > since some time now, parquet::ParquetFileWriter has the option to
> >> >> create
> >> >> > > buffered rowgroups with AppendBufferedRowGroup(), which basically
> >> >> gives
> >> >> > you
> >> >> > > the possibility to write to columns in any order you like (in
> >> >> contrast to
> >> >> > > the former only possible way of writing one column after the
> >> other).
> >> >> This
> >> >> > > is cool since it avoids the caller from having to create an in
> >> memory
> >> >> > > columnar representation of its data.
> >> >> > >
> >> >> > > However, when data size is huge compared to the available system
> >> >> memory
> >> >> > > (due to wide schema or a large rowgroupsize), this is
> problematic,
> >> as
> >> >> the
> >> >> > > buffers allocated internally can take up a large portion of RAM
> of
> >> the
> >> >> > > machine the conversion is running on.
> >> >> > >
> >> >> > > One way to solve that problem would be to use memory mapped files
> >> >> instead
> >> >> > > of plain memory buffers. That way, the number of required memory
> >> can
> >> >> be
> >> >> > > limited by the number of columns times the os-pagesize, which
> >> would be
> >> >> > > independent of the rowgroup-size. Consequently, large
> rowgroupsizes
> >> >> pose
> >> >> > no
> >> >> > > problem with respect to RAM consumption.
> >> >> > >
> >> >> > > I wonder what you generally think about the idea of integrating
> an
> >> >> > > AppendFileBufferedRowGroup() (or similar name) possibility which
> >> gives
> >> >> > the
> >> >> > > user the option to have the internal buffers be memory mapped
> >> files.
> >> >> > >
> >> >> > > After a quick look at how the buffers are managed inside arrow
> >> >> (allocated
> >> >> > > from a default memory pool), I have the impression that an
> >> >> implementation
> >> >> > > of this idea could be a rather huge change. I still wanted to
> know
> >> >> > whether
> >> >> > > that is something you could see being integrated or whether that
> is
> >> >> out
> >> >> > of
> >> >> > > scope of arrow.
> >> >> > >
> >> >> > > Thanks in advance and kind regards,
> >> >> > > Roman
> >> >> > >
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: Writing very large rowgroups to Apache Parquet
Posted by Jacques Nadeau <ja...@apache.org>.
I believe the formal Parquet standard already allows a file per column. At
least I remember it being discussed when the spec was first implemented. If
you look at the thrift spec it actually allows for this:
https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L771
That being said, I'm not sure which readers support this read pattern. If
it is part of the spec, doing it as a Parquet writing mode makes sense to
me.
On Mon, Jul 13, 2020 at 11:08 PM Roman Karlstetter <
roman.karlstetter@gmail.com> wrote:
> > I'd suggest a new write pattern. Write the columns page at a time to
> separate files then use a second process to concatenate the columns and
> append the footer. Odds are you would do better than os swapping and take
> memory requirements down to
> page size times field count.
>
> This is exactly what a student of us implemented pretty successfully:
> writing to one file per column (non-parquet, binary, memory-mapped). And
> once enough data is put into those "cache/buffer-files", the data is
> flushed to a parquet rowgroup.
>
> My question targeted the integration of these ideas into the arrow parquet
> writer. I wanted to know whether it makes sense to integrate these ideas or
> whether it's better to keep that functionality outside of arrow/parquet.
> Having it inside would have the benefit of reduced storage space because of
> encoding/compression and thus smaller overhead in the final copy phase
> (less data to copy and data already encoded/compressed). But on the other
> hand, having one memory mapped file per column is not something that seems
> to fit well with the current design of arrow.
>
> Thanks for the feedback,
> Roman
>
> Am So., 12. Juli 2020 um 03:05 Uhr schrieb Micah Kornfield <
> emkornfield@gmail.com>:
>
>> This is an interesting idea. For s3 multipart uploads one might run into
>> limitations pretty quickly (only 10k parts appear to be supported. all but
>> the last are expected to be at least 5mb if I read their docs correctly
>> [1])
>>
>> [1] https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html
>>
>>
>> On Saturday, July 11, 2020, Jacques Nadeau <ja...@apache.org> wrote:
>>
>> > I'd suggest a new write pattern. Write the columns page at a time to
>> > separate files then use a second process to concatenate the columns and
>> > append the footer. Odds are you would do better than os swapping and
>> take
>> > memory requirements down to page size times field count.
>> >
>> > In s3 I believe you could do this via a multipart upload and entirely
>> skip
>> > the second step. I don't know of any implementations that actually do
>> this
>> > yet.
>> >
>> > On Thu, Jul 9, 2020, 11:58 PM Roman Karlstetter <
>> > roman.karlstetter@gmail.com> wrote:
>> >
>> >> Hi,
>> >>
>> >> I wasn't aware of the fact that jemalloc mmap automatically for larger
>> >> allocations. And I didn't yet test this.
>> >>
>> >> The approach could be different in that we would know which parts of
>> the
>> >> buffers are going to be used next (the buffers are appendonly) and
>> which
>> >> parts won't be needed until actually flushing the rowgroup (and when
>> >> flushing, we also know the order). But I'm not sure whether that
>> knowledge
>> >> helps a lot in a) saving memory compared to a generic allocator or b)
>> >> improving performance. In addition to that, communicating this
>> knowledge
>> >> to
>> >> the implementation will also be tricky for the general case, I guess.
>> >>
>> >> Regarding setting the allocator to another memory pool: I was unsure
>> >> whether the memory pool is used for further allocations where the
>> default
>> >> memory pool would be more appropriate. If not, then setting the memory
>> >> pool
>> >> in the writer properties should actually work well.
>> >>
>> >> Maybe I should just play a bit with the different memory pool options
>> and
>> >> see how they behave. It makes more sense to discuss further ideas once
>> I
>> >> have some performance numbers.
>> >>
>> >> Thanks,
>> >> Roman
>> >>
>> >>
>> >> Am Fr., 10. Juli 2020 um 06:47 Uhr schrieb Micah Kornfield <
>> >> emkornfield@gmail.com>:
>> >>
>> >> > +parquet-dev as this seems more concerned with the non-arrow pieces
>> of
>> >> > parquet
>> >> >
>> >> > Hi Roman,
>> >> > Answers inline.
>> >> >
>> >> > One way to solve that problem would be to use memory mapped files
>> >> instead
>> >> > > of plain memory buffers. That way, the number of required memory
>> can
>> >> be
>> >> > > limited by the number of columns times the os-pagesize, which
>> would be
>> >> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
>> >> pose
>> >> > no
>> >> > > problem with respect to RAM consumption.
>> >> >
>> >> > I was under the impression that modern allocator (i.e. jemalloc)
>> already
>> >> > mmap for large allocations. How would this approach be different
>> from
>> >> the
>> >> > way allocators use it? Have you prototyped this approach to see if
>> it
>> >> > allows for better scalability?
>> >> >
>> >> >
>> >> > > After a quick look at how the buffers are managed inside arrow
>> >> (allocated
>> >> > > from a default memory pool), I have the impression that an
>> >> implementation
>> >> > > of this idea could be a rather huge change. I still wanted to know
>> >> > whether
>> >> > > that is something you could see being integrated or whether that is
>> >> out
>> >> > of
>> >> > > scope of arrow.
>> >> >
>> >> >
>> >> > A huge change probably isn't a great idea unless we've validated the
>> >> > approach along with alternatives. Is there currently code that
>> doesn't
>> >> > make use of the MemoryPool [1] provided by WriterProperties? If so we
>> >> > should probably fix it. Otherwise, is there a reason that you can't
>> >> > substitute a customized memory pool on WriterProperties?
>> >> >
>> >> > Thanks,
>> >> > Micah
>> >> >
>> >> > [1]
>> >> >
>> >> > https://github.com/apache/arrow/blob/5602c459eb8773b6be8059b1b11817
>> >> 5e9f16b7a3/cpp/src/parquet/properties.h#L447
>> >> >
>> >> > On Thu, Jul 9, 2020 at 8:35 AM Roman Karlstetter <
>> >> > roman.karlstetter@gmail.com> wrote:
>> >> >
>> >> > > Hi everyone,
>> >> > >
>> >> > > since some time now, parquet::ParquetFileWriter has the option to
>> >> create
>> >> > > buffered rowgroups with AppendBufferedRowGroup(), which basically
>> >> gives
>> >> > you
>> >> > > the possibility to write to columns in any order you like (in
>> >> contrast to
>> >> > > the former only possible way of writing one column after the
>> other).
>> >> This
>> >> > > is cool since it avoids the caller from having to create an in
>> memory
>> >> > > columnar representation of its data.
>> >> > >
>> >> > > However, when data size is huge compared to the available system
>> >> memory
>> >> > > (due to wide schema or a large rowgroupsize), this is problematic,
>> as
>> >> the
>> >> > > buffers allocated internally can take up a large portion of RAM of
>> the
>> >> > > machine the conversion is running on.
>> >> > >
>> >> > > One way to solve that problem would be to use memory mapped files
>> >> instead
>> >> > > of plain memory buffers. That way, the number of required memory
>> can
>> >> be
>> >> > > limited by the number of columns times the os-pagesize, which
>> would be
>> >> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
>> >> pose
>> >> > no
>> >> > > problem with respect to RAM consumption.
>> >> > >
>> >> > > I wonder what you generally think about the idea of integrating an
>> >> > > AppendFileBufferedRowGroup() (or similar name) possibility which
>> gives
>> >> > the
>> >> > > user the option to have the internal buffers be memory mapped
>> files.
>> >> > >
>> >> > > After a quick look at how the buffers are managed inside arrow
>> >> (allocated
>> >> > > from a default memory pool), I have the impression that an
>> >> implementation
>> >> > > of this idea could be a rather huge change. I still wanted to know
>> >> > whether
>> >> > > that is something you could see being integrated or whether that is
>> >> out
>> >> > of
>> >> > > scope of arrow.
>> >> > >
>> >> > > Thanks in advance and kind regards,
>> >> > > Roman
>> >> > >
>> >> >
>> >>
>> >
>>
>
Re: Writing very large rowgroups to Apache Parquet
Posted by Jacques Nadeau <ja...@apache.org>.
I believe the formal Parquet standard already allows a file per column. At
least I remember it being discussed when the spec was first implemented. If
you look at the thrift spec it actually allows for this:
https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L771
That being said, I'm not sure which readers support this read pattern. If
it is part of the spec, doing it as a Parquet writing mode makes sense to
me.
On Mon, Jul 13, 2020 at 11:08 PM Roman Karlstetter <
roman.karlstetter@gmail.com> wrote:
> > I'd suggest a new write pattern. Write the columns page at a time to
> separate files then use a second process to concatenate the columns and
> append the footer. Odds are you would do better than os swapping and take
> memory requirements down to
> page size times field count.
>
> This is exactly what a student of us implemented pretty successfully:
> writing to one file per column (non-parquet, binary, memory-mapped). And
> once enough data is put into those "cache/buffer-files", the data is
> flushed to a parquet rowgroup.
>
> My question targeted the integration of these ideas into the arrow parquet
> writer. I wanted to know whether it makes sense to integrate these ideas or
> whether it's better to keep that functionality outside of arrow/parquet.
> Having it inside would have the benefit of reduced storage space because of
> encoding/compression and thus smaller overhead in the final copy phase
> (less data to copy and data already encoded/compressed). But on the other
> hand, having one memory mapped file per column is not something that seems
> to fit well with the current design of arrow.
>
> Thanks for the feedback,
> Roman
>
> Am So., 12. Juli 2020 um 03:05 Uhr schrieb Micah Kornfield <
> emkornfield@gmail.com>:
>
>> This is an interesting idea. For s3 multipart uploads one might run into
>> limitations pretty quickly (only 10k parts appear to be supported. all but
>> the last are expected to be at least 5mb if I read their docs correctly
>> [1])
>>
>> [1] https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html
>>
>>
>> On Saturday, July 11, 2020, Jacques Nadeau <ja...@apache.org> wrote:
>>
>> > I'd suggest a new write pattern. Write the columns page at a time to
>> > separate files then use a second process to concatenate the columns and
>> > append the footer. Odds are you would do better than os swapping and
>> take
>> > memory requirements down to page size times field count.
>> >
>> > In s3 I believe you could do this via a multipart upload and entirely
>> skip
>> > the second step. I don't know of any implementations that actually do
>> this
>> > yet.
>> >
>> > On Thu, Jul 9, 2020, 11:58 PM Roman Karlstetter <
>> > roman.karlstetter@gmail.com> wrote:
>> >
>> >> Hi,
>> >>
>> >> I wasn't aware of the fact that jemalloc mmap automatically for larger
>> >> allocations. And I didn't yet test this.
>> >>
>> >> The approach could be different in that we would know which parts of
>> the
>> >> buffers are going to be used next (the buffers are appendonly) and
>> which
>> >> parts won't be needed until actually flushing the rowgroup (and when
>> >> flushing, we also know the order). But I'm not sure whether that
>> knowledge
>> >> helps a lot in a) saving memory compared to a generic allocator or b)
>> >> improving performance. In addition to that, communicating this
>> knowledge
>> >> to
>> >> the implementation will also be tricky for the general case, I guess.
>> >>
>> >> Regarding setting the allocator to another memory pool: I was unsure
>> >> whether the memory pool is used for further allocations where the
>> default
>> >> memory pool would be more appropriate. If not, then setting the memory
>> >> pool
>> >> in the writer properties should actually work well.
>> >>
>> >> Maybe I should just play a bit with the different memory pool options
>> and
>> >> see how they behave. It makes more sense to discuss further ideas once
>> I
>> >> have some performance numbers.
>> >>
>> >> Thanks,
>> >> Roman
>> >>
>> >>
>> >> Am Fr., 10. Juli 2020 um 06:47 Uhr schrieb Micah Kornfield <
>> >> emkornfield@gmail.com>:
>> >>
>> >> > +parquet-dev as this seems more concerned with the non-arrow pieces
>> of
>> >> > parquet
>> >> >
>> >> > Hi Roman,
>> >> > Answers inline.
>> >> >
>> >> > One way to solve that problem would be to use memory mapped files
>> >> instead
>> >> > > of plain memory buffers. That way, the number of required memory
>> can
>> >> be
>> >> > > limited by the number of columns times the os-pagesize, which
>> would be
>> >> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
>> >> pose
>> >> > no
>> >> > > problem with respect to RAM consumption.
>> >> >
>> >> > I was under the impression that modern allocator (i.e. jemalloc)
>> already
>> >> > mmap for large allocations. How would this approach be different
>> from
>> >> the
>> >> > way allocators use it? Have you prototyped this approach to see if
>> it
>> >> > allows for better scalability?
>> >> >
>> >> >
>> >> > > After a quick look at how the buffers are managed inside arrow
>> >> (allocated
>> >> > > from a default memory pool), I have the impression that an
>> >> implementation
>> >> > > of this idea could be a rather huge change. I still wanted to know
>> >> > whether
>> >> > > that is something you could see being integrated or whether that is
>> >> out
>> >> > of
>> >> > > scope of arrow.
>> >> >
>> >> >
>> >> > A huge change probably isn't a great idea unless we've validated the
>> >> > approach along with alternatives. Is there currently code that
>> doesn't
>> >> > make use of the MemoryPool [1] provided by WriterProperties? If so we
>> >> > should probably fix it. Otherwise, is there a reason that you can't
>> >> > substitute a customized memory pool on WriterProperties?
>> >> >
>> >> > Thanks,
>> >> > Micah
>> >> >
>> >> > [1]
>> >> >
>> >> > https://github.com/apache/arrow/blob/5602c459eb8773b6be8059b1b11817
>> >> 5e9f16b7a3/cpp/src/parquet/properties.h#L447
>> >> >
>> >> > On Thu, Jul 9, 2020 at 8:35 AM Roman Karlstetter <
>> >> > roman.karlstetter@gmail.com> wrote:
>> >> >
>> >> > > Hi everyone,
>> >> > >
>> >> > > since some time now, parquet::ParquetFileWriter has the option to
>> >> create
>> >> > > buffered rowgroups with AppendBufferedRowGroup(), which basically
>> >> gives
>> >> > you
>> >> > > the possibility to write to columns in any order you like (in
>> >> contrast to
>> >> > > the former only possible way of writing one column after the
>> other).
>> >> This
>> >> > > is cool since it avoids the caller from having to create an in
>> memory
>> >> > > columnar representation of its data.
>> >> > >
>> >> > > However, when data size is huge compared to the available system
>> >> memory
>> >> > > (due to wide schema or a large rowgroupsize), this is problematic,
>> as
>> >> the
>> >> > > buffers allocated internally can take up a large portion of RAM of
>> the
>> >> > > machine the conversion is running on.
>> >> > >
>> >> > > One way to solve that problem would be to use memory mapped files
>> >> instead
>> >> > > of plain memory buffers. That way, the number of required memory
>> can
>> >> be
>> >> > > limited by the number of columns times the os-pagesize, which
>> would be
>> >> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
>> >> pose
>> >> > no
>> >> > > problem with respect to RAM consumption.
>> >> > >
>> >> > > I wonder what you generally think about the idea of integrating an
>> >> > > AppendFileBufferedRowGroup() (or similar name) possibility which
>> gives
>> >> > the
>> >> > > user the option to have the internal buffers be memory mapped
>> files.
>> >> > >
>> >> > > After a quick look at how the buffers are managed inside arrow
>> >> (allocated
>> >> > > from a default memory pool), I have the impression that an
>> >> implementation
>> >> > > of this idea could be a rather huge change. I still wanted to know
>> >> > whether
>> >> > > that is something you could see being integrated or whether that is
>> >> out
>> >> > of
>> >> > > scope of arrow.
>> >> > >
>> >> > > Thanks in advance and kind regards,
>> >> > > Roman
>> >> > >
>> >> >
>> >>
>> >
>>
>
Re: Writing very large rowgroups to Apache Parquet
Posted by Roman Karlstetter <ro...@gmail.com>.
> I'd suggest a new write pattern. Write the columns page at a time to
separate files then use a second process to concatenate the columns and
append the footer. Odds are you would do better than os swapping and take
memory requirements down to
page size times field count.
This is exactly what a student of us implemented pretty successfully:
writing to one file per column (non-parquet, binary, memory-mapped). And
once enough data is put into those "cache/buffer-files", the data is
flushed to a parquet rowgroup.
My question targeted the integration of these ideas into the arrow parquet
writer. I wanted to know whether it makes sense to integrate these ideas or
whether it's better to keep that functionality outside of arrow/parquet.
Having it inside would have the benefit of reduced storage space because of
encoding/compression and thus smaller overhead in the final copy phase
(less data to copy and data already encoded/compressed). But on the other
hand, having one memory mapped file per column is not something that seems
to fit well with the current design of arrow.
Thanks for the feedback,
Roman
Am So., 12. Juli 2020 um 03:05 Uhr schrieb Micah Kornfield <
emkornfield@gmail.com>:
> This is an interesting idea. For s3 multipart uploads one might run into
> limitations pretty quickly (only 10k parts appear to be supported. all but
> the last are expected to be at least 5mb if I read their docs correctly
> [1])
>
> [1] https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html
>
>
> On Saturday, July 11, 2020, Jacques Nadeau <ja...@apache.org> wrote:
>
> > I'd suggest a new write pattern. Write the columns page at a time to
> > separate files then use a second process to concatenate the columns and
> > append the footer. Odds are you would do better than os swapping and take
> > memory requirements down to page size times field count.
> >
> > In s3 I believe you could do this via a multipart upload and entirely
> skip
> > the second step. I don't know of any implementations that actually do
> this
> > yet.
> >
> > On Thu, Jul 9, 2020, 11:58 PM Roman Karlstetter <
> > roman.karlstetter@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I wasn't aware of the fact that jemalloc mmap automatically for larger
> >> allocations. And I didn't yet test this.
> >>
> >> The approach could be different in that we would know which parts of the
> >> buffers are going to be used next (the buffers are appendonly) and which
> >> parts won't be needed until actually flushing the rowgroup (and when
> >> flushing, we also know the order). But I'm not sure whether that
> knowledge
> >> helps a lot in a) saving memory compared to a generic allocator or b)
> >> improving performance. In addition to that, communicating this knowledge
> >> to
> >> the implementation will also be tricky for the general case, I guess.
> >>
> >> Regarding setting the allocator to another memory pool: I was unsure
> >> whether the memory pool is used for further allocations where the
> default
> >> memory pool would be more appropriate. If not, then setting the memory
> >> pool
> >> in the writer properties should actually work well.
> >>
> >> Maybe I should just play a bit with the different memory pool options
> and
> >> see how they behave. It makes more sense to discuss further ideas once I
> >> have some performance numbers.
> >>
> >> Thanks,
> >> Roman
> >>
> >>
> >> Am Fr., 10. Juli 2020 um 06:47 Uhr schrieb Micah Kornfield <
> >> emkornfield@gmail.com>:
> >>
> >> > +parquet-dev as this seems more concerned with the non-arrow pieces of
> >> > parquet
> >> >
> >> > Hi Roman,
> >> > Answers inline.
> >> >
> >> > One way to solve that problem would be to use memory mapped files
> >> instead
> >> > > of plain memory buffers. That way, the number of required memory can
> >> be
> >> > > limited by the number of columns times the os-pagesize, which would
> be
> >> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
> >> pose
> >> > no
> >> > > problem with respect to RAM consumption.
> >> >
> >> > I was under the impression that modern allocator (i.e. jemalloc)
> already
> >> > mmap for large allocations. How would this approach be different from
> >> the
> >> > way allocators use it? Have you prototyped this approach to see if it
> >> > allows for better scalability?
> >> >
> >> >
> >> > > After a quick look at how the buffers are managed inside arrow
> >> (allocated
> >> > > from a default memory pool), I have the impression that an
> >> implementation
> >> > > of this idea could be a rather huge change. I still wanted to know
> >> > whether
> >> > > that is something you could see being integrated or whether that is
> >> out
> >> > of
> >> > > scope of arrow.
> >> >
> >> >
> >> > A huge change probably isn't a great idea unless we've validated the
> >> > approach along with alternatives. Is there currently code that
> doesn't
> >> > make use of the MemoryPool [1] provided by WriterProperties? If so we
> >> > should probably fix it. Otherwise, is there a reason that you can't
> >> > substitute a customized memory pool on WriterProperties?
> >> >
> >> > Thanks,
> >> > Micah
> >> >
> >> > [1]
> >> >
> >> > https://github.com/apache/arrow/blob/5602c459eb8773b6be8059b1b11817
> >> 5e9f16b7a3/cpp/src/parquet/properties.h#L447
> >> >
> >> > On Thu, Jul 9, 2020 at 8:35 AM Roman Karlstetter <
> >> > roman.karlstetter@gmail.com> wrote:
> >> >
> >> > > Hi everyone,
> >> > >
> >> > > since some time now, parquet::ParquetFileWriter has the option to
> >> create
> >> > > buffered rowgroups with AppendBufferedRowGroup(), which basically
> >> gives
> >> > you
> >> > > the possibility to write to columns in any order you like (in
> >> contrast to
> >> > > the former only possible way of writing one column after the other).
> >> This
> >> > > is cool since it avoids the caller from having to create an in
> memory
> >> > > columnar representation of its data.
> >> > >
> >> > > However, when data size is huge compared to the available system
> >> memory
> >> > > (due to wide schema or a large rowgroupsize), this is problematic,
> as
> >> the
> >> > > buffers allocated internally can take up a large portion of RAM of
> the
> >> > > machine the conversion is running on.
> >> > >
> >> > > One way to solve that problem would be to use memory mapped files
> >> instead
> >> > > of plain memory buffers. That way, the number of required memory can
> >> be
> >> > > limited by the number of columns times the os-pagesize, which would
> be
> >> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
> >> pose
> >> > no
> >> > > problem with respect to RAM consumption.
> >> > >
> >> > > I wonder what you generally think about the idea of integrating an
> >> > > AppendFileBufferedRowGroup() (or similar name) possibility which
> gives
> >> > the
> >> > > user the option to have the internal buffers be memory mapped files.
> >> > >
> >> > > After a quick look at how the buffers are managed inside arrow
> >> (allocated
> >> > > from a default memory pool), I have the impression that an
> >> implementation
> >> > > of this idea could be a rather huge change. I still wanted to know
> >> > whether
> >> > > that is something you could see being integrated or whether that is
> >> out
> >> > of
> >> > > scope of arrow.
> >> > >
> >> > > Thanks in advance and kind regards,
> >> > > Roman
> >> > >
> >> >
> >>
> >
>
Re: Writing very large rowgroups to Apache Parquet
Posted by Roman Karlstetter <ro...@gmail.com>.
> I'd suggest a new write pattern. Write the columns page at a time to
separate files then use a second process to concatenate the columns and
append the footer. Odds are you would do better than os swapping and take
memory requirements down to
page size times field count.
This is exactly what a student of us implemented pretty successfully:
writing to one file per column (non-parquet, binary, memory-mapped). And
once enough data is put into those "cache/buffer-files", the data is
flushed to a parquet rowgroup.
My question targeted the integration of these ideas into the arrow parquet
writer. I wanted to know whether it makes sense to integrate these ideas or
whether it's better to keep that functionality outside of arrow/parquet.
Having it inside would have the benefit of reduced storage space because of
encoding/compression and thus smaller overhead in the final copy phase
(less data to copy and data already encoded/compressed). But on the other
hand, having one memory mapped file per column is not something that seems
to fit well with the current design of arrow.
Thanks for the feedback,
Roman
Am So., 12. Juli 2020 um 03:05 Uhr schrieb Micah Kornfield <
emkornfield@gmail.com>:
> This is an interesting idea. For s3 multipart uploads one might run into
> limitations pretty quickly (only 10k parts appear to be supported. all but
> the last are expected to be at least 5mb if I read their docs correctly
> [1])
>
> [1] https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html
>
>
> On Saturday, July 11, 2020, Jacques Nadeau <ja...@apache.org> wrote:
>
> > I'd suggest a new write pattern. Write the columns page at a time to
> > separate files then use a second process to concatenate the columns and
> > append the footer. Odds are you would do better than os swapping and take
> > memory requirements down to page size times field count.
> >
> > In s3 I believe you could do this via a multipart upload and entirely
> skip
> > the second step. I don't know of any implementations that actually do
> this
> > yet.
> >
> > On Thu, Jul 9, 2020, 11:58 PM Roman Karlstetter <
> > roman.karlstetter@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I wasn't aware of the fact that jemalloc mmap automatically for larger
> >> allocations. And I didn't yet test this.
> >>
> >> The approach could be different in that we would know which parts of the
> >> buffers are going to be used next (the buffers are appendonly) and which
> >> parts won't be needed until actually flushing the rowgroup (and when
> >> flushing, we also know the order). But I'm not sure whether that
> knowledge
> >> helps a lot in a) saving memory compared to a generic allocator or b)
> >> improving performance. In addition to that, communicating this knowledge
> >> to
> >> the implementation will also be tricky for the general case, I guess.
> >>
> >> Regarding setting the allocator to another memory pool: I was unsure
> >> whether the memory pool is used for further allocations where the
> default
> >> memory pool would be more appropriate. If not, then setting the memory
> >> pool
> >> in the writer properties should actually work well.
> >>
> >> Maybe I should just play a bit with the different memory pool options
> and
> >> see how they behave. It makes more sense to discuss further ideas once I
> >> have some performance numbers.
> >>
> >> Thanks,
> >> Roman
> >>
> >>
> >> Am Fr., 10. Juli 2020 um 06:47 Uhr schrieb Micah Kornfield <
> >> emkornfield@gmail.com>:
> >>
> >> > +parquet-dev as this seems more concerned with the non-arrow pieces of
> >> > parquet
> >> >
> >> > Hi Roman,
> >> > Answers inline.
> >> >
> >> > One way to solve that problem would be to use memory mapped files
> >> instead
> >> > > of plain memory buffers. That way, the number of required memory can
> >> be
> >> > > limited by the number of columns times the os-pagesize, which would
> be
> >> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
> >> pose
> >> > no
> >> > > problem with respect to RAM consumption.
> >> >
> >> > I was under the impression that modern allocator (i.e. jemalloc)
> already
> >> > mmap for large allocations. How would this approach be different from
> >> the
> >> > way allocators use it? Have you prototyped this approach to see if it
> >> > allows for better scalability?
> >> >
> >> >
> >> > > After a quick look at how the buffers are managed inside arrow
> >> (allocated
> >> > > from a default memory pool), I have the impression that an
> >> implementation
> >> > > of this idea could be a rather huge change. I still wanted to know
> >> > whether
> >> > > that is something you could see being integrated or whether that is
> >> out
> >> > of
> >> > > scope of arrow.
> >> >
> >> >
> >> > A huge change probably isn't a great idea unless we've validated the
> >> > approach along with alternatives. Is there currently code that
> doesn't
> >> > make use of the MemoryPool [1] provided by WriterProperties? If so we
> >> > should probably fix it. Otherwise, is there a reason that you can't
> >> > substitute a customized memory pool on WriterProperties?
> >> >
> >> > Thanks,
> >> > Micah
> >> >
> >> > [1]
> >> >
> >> > https://github.com/apache/arrow/blob/5602c459eb8773b6be8059b1b11817
> >> 5e9f16b7a3/cpp/src/parquet/properties.h#L447
> >> >
> >> > On Thu, Jul 9, 2020 at 8:35 AM Roman Karlstetter <
> >> > roman.karlstetter@gmail.com> wrote:
> >> >
> >> > > Hi everyone,
> >> > >
> >> > > since some time now, parquet::ParquetFileWriter has the option to
> >> create
> >> > > buffered rowgroups with AppendBufferedRowGroup(), which basically
> >> gives
> >> > you
> >> > > the possibility to write to columns in any order you like (in
> >> contrast to
> >> > > the former only possible way of writing one column after the other).
> >> This
> >> > > is cool since it avoids the caller from having to create an in
> memory
> >> > > columnar representation of its data.
> >> > >
> >> > > However, when data size is huge compared to the available system
> >> memory
> >> > > (due to wide schema or a large rowgroupsize), this is problematic,
> as
> >> the
> >> > > buffers allocated internally can take up a large portion of RAM of
> the
> >> > > machine the conversion is running on.
> >> > >
> >> > > One way to solve that problem would be to use memory mapped files
> >> instead
> >> > > of plain memory buffers. That way, the number of required memory can
> >> be
> >> > > limited by the number of columns times the os-pagesize, which would
> be
> >> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
> >> pose
> >> > no
> >> > > problem with respect to RAM consumption.
> >> > >
> >> > > I wonder what you generally think about the idea of integrating an
> >> > > AppendFileBufferedRowGroup() (or similar name) possibility which
> gives
> >> > the
> >> > > user the option to have the internal buffers be memory mapped files.
> >> > >
> >> > > After a quick look at how the buffers are managed inside arrow
> >> (allocated
> >> > > from a default memory pool), I have the impression that an
> >> implementation
> >> > > of this idea could be a rather huge change. I still wanted to know
> >> > whether
> >> > > that is something you could see being integrated or whether that is
> >> out
> >> > of
> >> > > scope of arrow.
> >> > >
> >> > > Thanks in advance and kind regards,
> >> > > Roman
> >> > >
> >> >
> >>
> >
>
Re: Writing very large rowgroups to Apache Parquet
Posted by Micah Kornfield <em...@gmail.com>.
This is an interesting idea. For s3 multipart uploads one might run into
limitations pretty quickly (only 10k parts appear to be supported. all but
the last are expected to be at least 5mb if I read their docs correctly [1])
[1] https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html
On Saturday, July 11, 2020, Jacques Nadeau <ja...@apache.org> wrote:
> I'd suggest a new write pattern. Write the columns page at a time to
> separate files then use a second process to concatenate the columns and
> append the footer. Odds are you would do better than os swapping and take
> memory requirements down to page size times field count.
>
> In s3 I believe you could do this via a multipart upload and entirely skip
> the second step. I don't know of any implementations that actually do this
> yet.
>
> On Thu, Jul 9, 2020, 11:58 PM Roman Karlstetter <
> roman.karlstetter@gmail.com> wrote:
>
>> Hi,
>>
>> I wasn't aware of the fact that jemalloc mmap automatically for larger
>> allocations. And I didn't yet test this.
>>
>> The approach could be different in that we would know which parts of the
>> buffers are going to be used next (the buffers are appendonly) and which
>> parts won't be needed until actually flushing the rowgroup (and when
>> flushing, we also know the order). But I'm not sure whether that knowledge
>> helps a lot in a) saving memory compared to a generic allocator or b)
>> improving performance. In addition to that, communicating this knowledge
>> to
>> the implementation will also be tricky for the general case, I guess.
>>
>> Regarding setting the allocator to another memory pool: I was unsure
>> whether the memory pool is used for further allocations where the default
>> memory pool would be more appropriate. If not, then setting the memory
>> pool
>> in the writer properties should actually work well.
>>
>> Maybe I should just play a bit with the different memory pool options and
>> see how they behave. It makes more sense to discuss further ideas once I
>> have some performance numbers.
>>
>> Thanks,
>> Roman
>>
>>
>> Am Fr., 10. Juli 2020 um 06:47 Uhr schrieb Micah Kornfield <
>> emkornfield@gmail.com>:
>>
>> > +parquet-dev as this seems more concerned with the non-arrow pieces of
>> > parquet
>> >
>> > Hi Roman,
>> > Answers inline.
>> >
>> > One way to solve that problem would be to use memory mapped files
>> instead
>> > > of plain memory buffers. That way, the number of required memory can
>> be
>> > > limited by the number of columns times the os-pagesize, which would be
>> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
>> pose
>> > no
>> > > problem with respect to RAM consumption.
>> >
>> > I was under the impression that modern allocator (i.e. jemalloc) already
>> > mmap for large allocations. How would this approach be different from
>> the
>> > way allocators use it? Have you prototyped this approach to see if it
>> > allows for better scalability?
>> >
>> >
>> > > After a quick look at how the buffers are managed inside arrow
>> (allocated
>> > > from a default memory pool), I have the impression that an
>> implementation
>> > > of this idea could be a rather huge change. I still wanted to know
>> > whether
>> > > that is something you could see being integrated or whether that is
>> out
>> > of
>> > > scope of arrow.
>> >
>> >
>> > A huge change probably isn't a great idea unless we've validated the
>> > approach along with alternatives. Is there currently code that doesn't
>> > make use of the MemoryPool [1] provided by WriterProperties? If so we
>> > should probably fix it. Otherwise, is there a reason that you can't
>> > substitute a customized memory pool on WriterProperties?
>> >
>> > Thanks,
>> > Micah
>> >
>> > [1]
>> >
>> > https://github.com/apache/arrow/blob/5602c459eb8773b6be8059b1b11817
>> 5e9f16b7a3/cpp/src/parquet/properties.h#L447
>> >
>> > On Thu, Jul 9, 2020 at 8:35 AM Roman Karlstetter <
>> > roman.karlstetter@gmail.com> wrote:
>> >
>> > > Hi everyone,
>> > >
>> > > since some time now, parquet::ParquetFileWriter has the option to
>> create
>> > > buffered rowgroups with AppendBufferedRowGroup(), which basically
>> gives
>> > you
>> > > the possibility to write to columns in any order you like (in
>> contrast to
>> > > the former only possible way of writing one column after the other).
>> This
>> > > is cool since it avoids the caller from having to create an in memory
>> > > columnar representation of its data.
>> > >
>> > > However, when data size is huge compared to the available system
>> memory
>> > > (due to wide schema or a large rowgroupsize), this is problematic, as
>> the
>> > > buffers allocated internally can take up a large portion of RAM of the
>> > > machine the conversion is running on.
>> > >
>> > > One way to solve that problem would be to use memory mapped files
>> instead
>> > > of plain memory buffers. That way, the number of required memory can
>> be
>> > > limited by the number of columns times the os-pagesize, which would be
>> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
>> pose
>> > no
>> > > problem with respect to RAM consumption.
>> > >
>> > > I wonder what you generally think about the idea of integrating an
>> > > AppendFileBufferedRowGroup() (or similar name) possibility which gives
>> > the
>> > > user the option to have the internal buffers be memory mapped files.
>> > >
>> > > After a quick look at how the buffers are managed inside arrow
>> (allocated
>> > > from a default memory pool), I have the impression that an
>> implementation
>> > > of this idea could be a rather huge change. I still wanted to know
>> > whether
>> > > that is something you could see being integrated or whether that is
>> out
>> > of
>> > > scope of arrow.
>> > >
>> > > Thanks in advance and kind regards,
>> > > Roman
>> > >
>> >
>>
>
Re: Writing very large rowgroups to Apache Parquet
Posted by Micah Kornfield <em...@gmail.com>.
This is an interesting idea. For s3 multipart uploads one might run into
limitations pretty quickly (only 10k parts appear to be supported. all but
the last are expected to be at least 5mb if I read their docs correctly [1])
[1] https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html
On Saturday, July 11, 2020, Jacques Nadeau <ja...@apache.org> wrote:
> I'd suggest a new write pattern. Write the columns page at a time to
> separate files then use a second process to concatenate the columns and
> append the footer. Odds are you would do better than os swapping and take
> memory requirements down to page size times field count.
>
> In s3 I believe you could do this via a multipart upload and entirely skip
> the second step. I don't know of any implementations that actually do this
> yet.
>
> On Thu, Jul 9, 2020, 11:58 PM Roman Karlstetter <
> roman.karlstetter@gmail.com> wrote:
>
>> Hi,
>>
>> I wasn't aware of the fact that jemalloc mmap automatically for larger
>> allocations. And I didn't yet test this.
>>
>> The approach could be different in that we would know which parts of the
>> buffers are going to be used next (the buffers are appendonly) and which
>> parts won't be needed until actually flushing the rowgroup (and when
>> flushing, we also know the order). But I'm not sure whether that knowledge
>> helps a lot in a) saving memory compared to a generic allocator or b)
>> improving performance. In addition to that, communicating this knowledge
>> to
>> the implementation will also be tricky for the general case, I guess.
>>
>> Regarding setting the allocator to another memory pool: I was unsure
>> whether the memory pool is used for further allocations where the default
>> memory pool would be more appropriate. If not, then setting the memory
>> pool
>> in the writer properties should actually work well.
>>
>> Maybe I should just play a bit with the different memory pool options and
>> see how they behave. It makes more sense to discuss further ideas once I
>> have some performance numbers.
>>
>> Thanks,
>> Roman
>>
>>
>> Am Fr., 10. Juli 2020 um 06:47 Uhr schrieb Micah Kornfield <
>> emkornfield@gmail.com>:
>>
>> > +parquet-dev as this seems more concerned with the non-arrow pieces of
>> > parquet
>> >
>> > Hi Roman,
>> > Answers inline.
>> >
>> > One way to solve that problem would be to use memory mapped files
>> instead
>> > > of plain memory buffers. That way, the number of required memory can
>> be
>> > > limited by the number of columns times the os-pagesize, which would be
>> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
>> pose
>> > no
>> > > problem with respect to RAM consumption.
>> >
>> > I was under the impression that modern allocator (i.e. jemalloc) already
>> > mmap for large allocations. How would this approach be different from
>> the
>> > way allocators use it? Have you prototyped this approach to see if it
>> > allows for better scalability?
>> >
>> >
>> > > After a quick look at how the buffers are managed inside arrow
>> (allocated
>> > > from a default memory pool), I have the impression that an
>> implementation
>> > > of this idea could be a rather huge change. I still wanted to know
>> > whether
>> > > that is something you could see being integrated or whether that is
>> out
>> > of
>> > > scope of arrow.
>> >
>> >
>> > A huge change probably isn't a great idea unless we've validated the
>> > approach along with alternatives. Is there currently code that doesn't
>> > make use of the MemoryPool [1] provided by WriterProperties? If so we
>> > should probably fix it. Otherwise, is there a reason that you can't
>> > substitute a customized memory pool on WriterProperties?
>> >
>> > Thanks,
>> > Micah
>> >
>> > [1]
>> >
>> > https://github.com/apache/arrow/blob/5602c459eb8773b6be8059b1b11817
>> 5e9f16b7a3/cpp/src/parquet/properties.h#L447
>> >
>> > On Thu, Jul 9, 2020 at 8:35 AM Roman Karlstetter <
>> > roman.karlstetter@gmail.com> wrote:
>> >
>> > > Hi everyone,
>> > >
>> > > since some time now, parquet::ParquetFileWriter has the option to
>> create
>> > > buffered rowgroups with AppendBufferedRowGroup(), which basically
>> gives
>> > you
>> > > the possibility to write to columns in any order you like (in
>> contrast to
>> > > the former only possible way of writing one column after the other).
>> This
>> > > is cool since it avoids the caller from having to create an in memory
>> > > columnar representation of its data.
>> > >
>> > > However, when data size is huge compared to the available system
>> memory
>> > > (due to wide schema or a large rowgroupsize), this is problematic, as
>> the
>> > > buffers allocated internally can take up a large portion of RAM of the
>> > > machine the conversion is running on.
>> > >
>> > > One way to solve that problem would be to use memory mapped files
>> instead
>> > > of plain memory buffers. That way, the number of required memory can
>> be
>> > > limited by the number of columns times the os-pagesize, which would be
>> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
>> pose
>> > no
>> > > problem with respect to RAM consumption.
>> > >
>> > > I wonder what you generally think about the idea of integrating an
>> > > AppendFileBufferedRowGroup() (or similar name) possibility which gives
>> > the
>> > > user the option to have the internal buffers be memory mapped files.
>> > >
>> > > After a quick look at how the buffers are managed inside arrow
>> (allocated
>> > > from a default memory pool), I have the impression that an
>> implementation
>> > > of this idea could be a rather huge change. I still wanted to know
>> > whether
>> > > that is something you could see being integrated or whether that is
>> out
>> > of
>> > > scope of arrow.
>> > >
>> > > Thanks in advance and kind regards,
>> > > Roman
>> > >
>> >
>>
>
Re: Writing very large rowgroups to Apache Parquet
Posted by Jacques Nadeau <ja...@apache.org>.
I'd suggest a new write pattern. Write the columns page at a time to
separate files then use a second process to concatenate the columns and
append the footer. Odds are you would do better than os swapping and take
memory requirements down to page size times field count.
In s3 I believe you could do this via a multipart upload and entirely skip
the second step. I don't know of any implementations that actually do this
yet.
On Thu, Jul 9, 2020, 11:58 PM Roman Karlstetter <ro...@gmail.com>
wrote:
> Hi,
>
> I wasn't aware of the fact that jemalloc mmap automatically for larger
> allocations. And I didn't yet test this.
>
> The approach could be different in that we would know which parts of the
> buffers are going to be used next (the buffers are appendonly) and which
> parts won't be needed until actually flushing the rowgroup (and when
> flushing, we also know the order). But I'm not sure whether that knowledge
> helps a lot in a) saving memory compared to a generic allocator or b)
> improving performance. In addition to that, communicating this knowledge to
> the implementation will also be tricky for the general case, I guess.
>
> Regarding setting the allocator to another memory pool: I was unsure
> whether the memory pool is used for further allocations where the default
> memory pool would be more appropriate. If not, then setting the memory pool
> in the writer properties should actually work well.
>
> Maybe I should just play a bit with the different memory pool options and
> see how they behave. It makes more sense to discuss further ideas once I
> have some performance numbers.
>
> Thanks,
> Roman
>
>
> Am Fr., 10. Juli 2020 um 06:47 Uhr schrieb Micah Kornfield <
> emkornfield@gmail.com>:
>
> > +parquet-dev as this seems more concerned with the non-arrow pieces of
> > parquet
> >
> > Hi Roman,
> > Answers inline.
> >
> > One way to solve that problem would be to use memory mapped files instead
> > > of plain memory buffers. That way, the number of required memory can be
> > > limited by the number of columns times the os-pagesize, which would be
> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
> pose
> > no
> > > problem with respect to RAM consumption.
> >
> > I was under the impression that modern allocator (i.e. jemalloc) already
> > mmap for large allocations. How would this approach be different from
> the
> > way allocators use it? Have you prototyped this approach to see if it
> > allows for better scalability?
> >
> >
> > > After a quick look at how the buffers are managed inside arrow
> (allocated
> > > from a default memory pool), I have the impression that an
> implementation
> > > of this idea could be a rather huge change. I still wanted to know
> > whether
> > > that is something you could see being integrated or whether that is out
> > of
> > > scope of arrow.
> >
> >
> > A huge change probably isn't a great idea unless we've validated the
> > approach along with alternatives. Is there currently code that doesn't
> > make use of the MemoryPool [1] provided by WriterProperties? If so we
> > should probably fix it. Otherwise, is there a reason that you can't
> > substitute a customized memory pool on WriterProperties?
> >
> > Thanks,
> > Micah
> >
> > [1]
> >
> >
> https://github.com/apache/arrow/blob/5602c459eb8773b6be8059b1b118175e9f16b7a3/cpp/src/parquet/properties.h#L447
> >
> > On Thu, Jul 9, 2020 at 8:35 AM Roman Karlstetter <
> > roman.karlstetter@gmail.com> wrote:
> >
> > > Hi everyone,
> > >
> > > since some time now, parquet::ParquetFileWriter has the option to
> create
> > > buffered rowgroups with AppendBufferedRowGroup(), which basically gives
> > you
> > > the possibility to write to columns in any order you like (in contrast
> to
> > > the former only possible way of writing one column after the other).
> This
> > > is cool since it avoids the caller from having to create an in memory
> > > columnar representation of its data.
> > >
> > > However, when data size is huge compared to the available system memory
> > > (due to wide schema or a large rowgroupsize), this is problematic, as
> the
> > > buffers allocated internally can take up a large portion of RAM of the
> > > machine the conversion is running on.
> > >
> > > One way to solve that problem would be to use memory mapped files
> instead
> > > of plain memory buffers. That way, the number of required memory can be
> > > limited by the number of columns times the os-pagesize, which would be
> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
> pose
> > no
> > > problem with respect to RAM consumption.
> > >
> > > I wonder what you generally think about the idea of integrating an
> > > AppendFileBufferedRowGroup() (or similar name) possibility which gives
> > the
> > > user the option to have the internal buffers be memory mapped files.
> > >
> > > After a quick look at how the buffers are managed inside arrow
> (allocated
> > > from a default memory pool), I have the impression that an
> implementation
> > > of this idea could be a rather huge change. I still wanted to know
> > whether
> > > that is something you could see being integrated or whether that is out
> > of
> > > scope of arrow.
> > >
> > > Thanks in advance and kind regards,
> > > Roman
> > >
> >
>
Re: Writing very large rowgroups to Apache Parquet
Posted by Jacques Nadeau <ja...@apache.org>.
I'd suggest a new write pattern. Write the columns page at a time to
separate files then use a second process to concatenate the columns and
append the footer. Odds are you would do better than os swapping and take
memory requirements down to page size times field count.
In s3 I believe you could do this via a multipart upload and entirely skip
the second step. I don't know of any implementations that actually do this
yet.
On Thu, Jul 9, 2020, 11:58 PM Roman Karlstetter <ro...@gmail.com>
wrote:
> Hi,
>
> I wasn't aware of the fact that jemalloc mmap automatically for larger
> allocations. And I didn't yet test this.
>
> The approach could be different in that we would know which parts of the
> buffers are going to be used next (the buffers are appendonly) and which
> parts won't be needed until actually flushing the rowgroup (and when
> flushing, we also know the order). But I'm not sure whether that knowledge
> helps a lot in a) saving memory compared to a generic allocator or b)
> improving performance. In addition to that, communicating this knowledge to
> the implementation will also be tricky for the general case, I guess.
>
> Regarding setting the allocator to another memory pool: I was unsure
> whether the memory pool is used for further allocations where the default
> memory pool would be more appropriate. If not, then setting the memory pool
> in the writer properties should actually work well.
>
> Maybe I should just play a bit with the different memory pool options and
> see how they behave. It makes more sense to discuss further ideas once I
> have some performance numbers.
>
> Thanks,
> Roman
>
>
> Am Fr., 10. Juli 2020 um 06:47 Uhr schrieb Micah Kornfield <
> emkornfield@gmail.com>:
>
> > +parquet-dev as this seems more concerned with the non-arrow pieces of
> > parquet
> >
> > Hi Roman,
> > Answers inline.
> >
> > One way to solve that problem would be to use memory mapped files instead
> > > of plain memory buffers. That way, the number of required memory can be
> > > limited by the number of columns times the os-pagesize, which would be
> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
> pose
> > no
> > > problem with respect to RAM consumption.
> >
> > I was under the impression that modern allocator (i.e. jemalloc) already
> > mmap for large allocations. How would this approach be different from
> the
> > way allocators use it? Have you prototyped this approach to see if it
> > allows for better scalability?
> >
> >
> > > After a quick look at how the buffers are managed inside arrow
> (allocated
> > > from a default memory pool), I have the impression that an
> implementation
> > > of this idea could be a rather huge change. I still wanted to know
> > whether
> > > that is something you could see being integrated or whether that is out
> > of
> > > scope of arrow.
> >
> >
> > A huge change probably isn't a great idea unless we've validated the
> > approach along with alternatives. Is there currently code that doesn't
> > make use of the MemoryPool [1] provided by WriterProperties? If so we
> > should probably fix it. Otherwise, is there a reason that you can't
> > substitute a customized memory pool on WriterProperties?
> >
> > Thanks,
> > Micah
> >
> > [1]
> >
> >
> https://github.com/apache/arrow/blob/5602c459eb8773b6be8059b1b118175e9f16b7a3/cpp/src/parquet/properties.h#L447
> >
> > On Thu, Jul 9, 2020 at 8:35 AM Roman Karlstetter <
> > roman.karlstetter@gmail.com> wrote:
> >
> > > Hi everyone,
> > >
> > > since some time now, parquet::ParquetFileWriter has the option to
> create
> > > buffered rowgroups with AppendBufferedRowGroup(), which basically gives
> > you
> > > the possibility to write to columns in any order you like (in contrast
> to
> > > the former only possible way of writing one column after the other).
> This
> > > is cool since it avoids the caller from having to create an in memory
> > > columnar representation of its data.
> > >
> > > However, when data size is huge compared to the available system memory
> > > (due to wide schema or a large rowgroupsize), this is problematic, as
> the
> > > buffers allocated internally can take up a large portion of RAM of the
> > > machine the conversion is running on.
> > >
> > > One way to solve that problem would be to use memory mapped files
> instead
> > > of plain memory buffers. That way, the number of required memory can be
> > > limited by the number of columns times the os-pagesize, which would be
> > > independent of the rowgroup-size. Consequently, large rowgroupsizes
> pose
> > no
> > > problem with respect to RAM consumption.
> > >
> > > I wonder what you generally think about the idea of integrating an
> > > AppendFileBufferedRowGroup() (or similar name) possibility which gives
> > the
> > > user the option to have the internal buffers be memory mapped files.
> > >
> > > After a quick look at how the buffers are managed inside arrow
> (allocated
> > > from a default memory pool), I have the impression that an
> implementation
> > > of this idea could be a rather huge change. I still wanted to know
> > whether
> > > that is something you could see being integrated or whether that is out
> > of
> > > scope of arrow.
> > >
> > > Thanks in advance and kind regards,
> > > Roman
> > >
> >
>
Re: Writing very large rowgroups to Apache Parquet
Posted by Roman Karlstetter <ro...@gmail.com>.
Hi,
I wasn't aware of the fact that jemalloc mmap automatically for larger
allocations. And I didn't yet test this.
The approach could be different in that we would know which parts of the
buffers are going to be used next (the buffers are appendonly) and which
parts won't be needed until actually flushing the rowgroup (and when
flushing, we also know the order). But I'm not sure whether that knowledge
helps a lot in a) saving memory compared to a generic allocator or b)
improving performance. In addition to that, communicating this knowledge to
the implementation will also be tricky for the general case, I guess.
Regarding setting the allocator to another memory pool: I was unsure
whether the memory pool is used for further allocations where the default
memory pool would be more appropriate. If not, then setting the memory pool
in the writer properties should actually work well.
Maybe I should just play a bit with the different memory pool options and
see how they behave. It makes more sense to discuss further ideas once I
have some performance numbers.
Thanks,
Roman
Am Fr., 10. Juli 2020 um 06:47 Uhr schrieb Micah Kornfield <
emkornfield@gmail.com>:
> +parquet-dev as this seems more concerned with the non-arrow pieces of
> parquet
>
> Hi Roman,
> Answers inline.
>
> One way to solve that problem would be to use memory mapped files instead
> > of plain memory buffers. That way, the number of required memory can be
> > limited by the number of columns times the os-pagesize, which would be
> > independent of the rowgroup-size. Consequently, large rowgroupsizes pose
> no
> > problem with respect to RAM consumption.
>
> I was under the impression that modern allocator (i.e. jemalloc) already
> mmap for large allocations. How would this approach be different from the
> way allocators use it? Have you prototyped this approach to see if it
> allows for better scalability?
>
>
> > After a quick look at how the buffers are managed inside arrow (allocated
> > from a default memory pool), I have the impression that an implementation
> > of this idea could be a rather huge change. I still wanted to know
> whether
> > that is something you could see being integrated or whether that is out
> of
> > scope of arrow.
>
>
> A huge change probably isn't a great idea unless we've validated the
> approach along with alternatives. Is there currently code that doesn't
> make use of the MemoryPool [1] provided by WriterProperties? If so we
> should probably fix it. Otherwise, is there a reason that you can't
> substitute a customized memory pool on WriterProperties?
>
> Thanks,
> Micah
>
> [1]
>
> https://github.com/apache/arrow/blob/5602c459eb8773b6be8059b1b118175e9f16b7a3/cpp/src/parquet/properties.h#L447
>
> On Thu, Jul 9, 2020 at 8:35 AM Roman Karlstetter <
> roman.karlstetter@gmail.com> wrote:
>
> > Hi everyone,
> >
> > since some time now, parquet::ParquetFileWriter has the option to create
> > buffered rowgroups with AppendBufferedRowGroup(), which basically gives
> you
> > the possibility to write to columns in any order you like (in contrast to
> > the former only possible way of writing one column after the other). This
> > is cool since it avoids the caller from having to create an in memory
> > columnar representation of its data.
> >
> > However, when data size is huge compared to the available system memory
> > (due to wide schema or a large rowgroupsize), this is problematic, as the
> > buffers allocated internally can take up a large portion of RAM of the
> > machine the conversion is running on.
> >
> > One way to solve that problem would be to use memory mapped files instead
> > of plain memory buffers. That way, the number of required memory can be
> > limited by the number of columns times the os-pagesize, which would be
> > independent of the rowgroup-size. Consequently, large rowgroupsizes pose
> no
> > problem with respect to RAM consumption.
> >
> > I wonder what you generally think about the idea of integrating an
> > AppendFileBufferedRowGroup() (or similar name) possibility which gives
> the
> > user the option to have the internal buffers be memory mapped files.
> >
> > After a quick look at how the buffers are managed inside arrow (allocated
> > from a default memory pool), I have the impression that an implementation
> > of this idea could be a rather huge change. I still wanted to know
> whether
> > that is something you could see being integrated or whether that is out
> of
> > scope of arrow.
> >
> > Thanks in advance and kind regards,
> > Roman
> >
>
Re: Writing very large rowgroups to Apache Parquet
Posted by Roman Karlstetter <ro...@gmail.com>.
Hi,
I wasn't aware of the fact that jemalloc mmap automatically for larger
allocations. And I didn't yet test this.
The approach could be different in that we would know which parts of the
buffers are going to be used next (the buffers are appendonly) and which
parts won't be needed until actually flushing the rowgroup (and when
flushing, we also know the order). But I'm not sure whether that knowledge
helps a lot in a) saving memory compared to a generic allocator or b)
improving performance. In addition to that, communicating this knowledge to
the implementation will also be tricky for the general case, I guess.
Regarding setting the allocator to another memory pool: I was unsure
whether the memory pool is used for further allocations where the default
memory pool would be more appropriate. If not, then setting the memory pool
in the writer properties should actually work well.
Maybe I should just play a bit with the different memory pool options and
see how they behave. It makes more sense to discuss further ideas once I
have some performance numbers.
Thanks,
Roman
Am Fr., 10. Juli 2020 um 06:47 Uhr schrieb Micah Kornfield <
emkornfield@gmail.com>:
> +parquet-dev as this seems more concerned with the non-arrow pieces of
> parquet
>
> Hi Roman,
> Answers inline.
>
> One way to solve that problem would be to use memory mapped files instead
> > of plain memory buffers. That way, the number of required memory can be
> > limited by the number of columns times the os-pagesize, which would be
> > independent of the rowgroup-size. Consequently, large rowgroupsizes pose
> no
> > problem with respect to RAM consumption.
>
> I was under the impression that modern allocator (i.e. jemalloc) already
> mmap for large allocations. How would this approach be different from the
> way allocators use it? Have you prototyped this approach to see if it
> allows for better scalability?
>
>
> > After a quick look at how the buffers are managed inside arrow (allocated
> > from a default memory pool), I have the impression that an implementation
> > of this idea could be a rather huge change. I still wanted to know
> whether
> > that is something you could see being integrated or whether that is out
> of
> > scope of arrow.
>
>
> A huge change probably isn't a great idea unless we've validated the
> approach along with alternatives. Is there currently code that doesn't
> make use of the MemoryPool [1] provided by WriterProperties? If so we
> should probably fix it. Otherwise, is there a reason that you can't
> substitute a customized memory pool on WriterProperties?
>
> Thanks,
> Micah
>
> [1]
>
> https://github.com/apache/arrow/blob/5602c459eb8773b6be8059b1b118175e9f16b7a3/cpp/src/parquet/properties.h#L447
>
> On Thu, Jul 9, 2020 at 8:35 AM Roman Karlstetter <
> roman.karlstetter@gmail.com> wrote:
>
> > Hi everyone,
> >
> > since some time now, parquet::ParquetFileWriter has the option to create
> > buffered rowgroups with AppendBufferedRowGroup(), which basically gives
> you
> > the possibility to write to columns in any order you like (in contrast to
> > the former only possible way of writing one column after the other). This
> > is cool since it avoids the caller from having to create an in memory
> > columnar representation of its data.
> >
> > However, when data size is huge compared to the available system memory
> > (due to wide schema or a large rowgroupsize), this is problematic, as the
> > buffers allocated internally can take up a large portion of RAM of the
> > machine the conversion is running on.
> >
> > One way to solve that problem would be to use memory mapped files instead
> > of plain memory buffers. That way, the number of required memory can be
> > limited by the number of columns times the os-pagesize, which would be
> > independent of the rowgroup-size. Consequently, large rowgroupsizes pose
> no
> > problem with respect to RAM consumption.
> >
> > I wonder what you generally think about the idea of integrating an
> > AppendFileBufferedRowGroup() (or similar name) possibility which gives
> the
> > user the option to have the internal buffers be memory mapped files.
> >
> > After a quick look at how the buffers are managed inside arrow (allocated
> > from a default memory pool), I have the impression that an implementation
> > of this idea could be a rather huge change. I still wanted to know
> whether
> > that is something you could see being integrated or whether that is out
> of
> > scope of arrow.
> >
> > Thanks in advance and kind regards,
> > Roman
> >
>