Large data sets

[Shawn Willden <sh...@willden.org>]

Can thrift operate in a space-efficient manner on large data sets?
I'd like to generate and parse "messages" that are often hundreds of
megabytes in size, composed of a header struct followed by a long list
of structs.  When I parse the messages I will process the pieces in
order, but I need to know if the code will try to load the entire
message contents into RAM at once.  Since the documentation says that
list<type> is translated into an STL vector, I'm assuming it will.

Thanks,

--
Shawn

Re: Large data sets

[Shawn Willden <sh...@willden.org>]

On Mon, Feb 7, 2011 at 12:30 PM, Bryan Duxbury <br...@rapleaf.com> wrote:

> If you serialize your messages as a struct containing a list that has all
> the "hundreds of megabytes", then yes. If you're looking for something
> that's more iterable, you should instead write all of the components of the
> message separately and manage the list-ness of it on your own.


Thanks.  That's what I needed to know -- and what I expected.

-- 
Shawn

Re: Large data sets

[Bryan Duxbury <br...@rapleaf.com>]

If you serialize your messages as a struct containing a list that has all
the "hundreds of megabytes", then yes. If you're looking for something
that's more iterable, you should instead write all of the components of the
message separately and manage the list-ness of it on your own.

On Mon, Feb 7, 2011 at 10:21 AM, Shawn Willden <sh...@willden.org> wrote:

> Can thrift operate in a space-efficient manner on large data sets?
> I'd like to generate and parse "messages" that are often hundreds of
> megabytes in size, composed of a header struct followed by a long list
> of structs.  When I parse the messages I will process the pieces in
> order, but I need to know if the code will try to load the entire
> message contents into RAM at once.  Since the documentation says that
> list<type> is translated into an STL vector, I'm assuming it will.
>
> Thanks,
>
> --
> Shawn
>