You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Sergey Khenkin <sk...@gmail.com> on 2005/08/09 17:58:47 UTC
Very big arrays in Java and Commons Collections
Hi All,
Does anybody know of a mean to work with very large arrays that don't
fit in memory and are cahced to disk when necessary?
Is there a proper type of collection in Commons Collection?
Thanks.
--
Best regards,
Sergey Khenkin mailto:skhenkin@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: Very big arrays in Java and Commons Collections
Posted by Elifarley Callado Coelho Cruz <el...@gmail.com>.
Or you could use a BDB database:
http://today.java.net/pub/a/today/2004/08/24/sleepy.html
http://www.sleepycat.com/products/je.shtml
Excerpt:
"
Of course, when I go through this list, and try to explain Berkeley DB
to people, they usually respond with comments like:
So, it's, like, what? A well-behaved HashMap?
To which I usually reply, "well, yeah, sort of. It's actually a little
more like a well-behaved TreeMap, but you get the idea."
At its core, a Berkeley DB database is really just a b-tree with a
persistent file format and nice transactional semantics thrown in. It
makes sense to use it, or something like it, in situations when all of
the following are true:
* You need to store objects in an indexed collection.
* You only need a small number of fairly simple ways to retrieve the objects.
* You don't need ad-hoc queries.
* You need transactional insert/remove/update operations on your storage.
* You need persistence to disk.
* You need a very small footprint, and for the database to be in-process.
* You don't need to have multiple processes or applications access
the database.
"
Cheers,
Elifarley
On 8/9/05, Wade Chandler <hw...@yahoo.com> wrote:
> This sounds like a really good project proposal
> actually for a Collections based file mapping library
> or a large ArrayList...obviously unable to have a
> toArray method except for sections of the file.
> Anyways, I don't know of any direct wrappers for this,
> but you can use java.nio.FileChannel obtaining it from
> a java.io.RandomAccess file and then using the map
> method to map regions of the file into memory. This
> would allow you to read and write bytes, floats,
> doubles, ints, arrays of all etc to this mapped region
> making a larger area obviously than allowed memory
> without using much or hardly any depending on the
> needed efficiency concept. You could make use of this
> to store objects as well depending on how you are
> storing them. You can store them as arrays of bytes
> and read and write them out to the mapped area giving
> you more space to work....all conceptual of course.
> Might help you though.
>
> Wade
>
> --- Sergey Khenkin <sk...@gmail.com> wrote:
>
> > Hi All,
> >
> > Does anybody know of a mean to work with very large
> > arrays that don't
> > fit in memory and are cahced to disk when necessary?
> > Is there a proper type of collection in Commons
> > Collection?
> > Thanks.
> >
> > --
> > Best regards,
> > Sergey Khenkin
> > mailto:skhenkin@gmail.com
> >
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> > commons-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> > commons-user-help@jakarta.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: Very big arrays in Java and Commons Collections
Posted by Torsten Curdt <tc...@apache.org>.
On 09.08.2005, at 20:10, Wade Chandler wrote:
> This sounds like a really good project proposal
> actually for a Collections based file mapping library
> or a large ArrayList...obviously unable to have a
> toArray method except for sections of the file.
> Anyways, I don't know of any direct wrappers for this,
> but you can use java.nio.FileChannel obtaining it from
> a java.io.RandomAccess file and then using the map
> method to map regions of the file into memory. This
> would allow you to read and write bytes, floats,
> doubles, ints, arrays of all etc to this mapped region
> making a larger area obviously than allowed memory
> without using much or hardly any depending on the
> needed efficiency concept. You could make use of this
> to store objects as well depending on how you are
> storing them. You can store them as arrays of bytes
> and read and write them out to the mapped area giving
> you more space to work....all conceptual of course.
> Might help you though.
Yepp ...that would be awesome to have!
cheers
--
Torsten
Re: Very big arrays in Java and Commons Collections
Posted by Wade Chandler <hw...@yahoo.com>.
This sounds like a really good project proposal
actually for a Collections based file mapping library
or a large ArrayList...obviously unable to have a
toArray method except for sections of the file.
Anyways, I don't know of any direct wrappers for this,
but you can use java.nio.FileChannel obtaining it from
a java.io.RandomAccess file and then using the map
method to map regions of the file into memory. This
would allow you to read and write bytes, floats,
doubles, ints, arrays of all etc to this mapped region
making a larger area obviously than allowed memory
without using much or hardly any depending on the
needed efficiency concept. You could make use of this
to store objects as well depending on how you are
storing them. You can store them as arrays of bytes
and read and write them out to the mapped area giving
you more space to work....all conceptual of course.
Might help you though.
Wade
--- Sergey Khenkin <sk...@gmail.com> wrote:
> Hi All,
>
> Does anybody know of a mean to work with very large
> arrays that don't
> fit in memory and are cahced to disk when necessary?
> Is there a proper type of collection in Commons
> Collection?
> Thanks.
>
> --
> Best regards,
> Sergey Khenkin
> mailto:skhenkin@gmail.com
>
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> commons-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org