You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Leon Rosenberg <ro...@googlemail.com> on 2009/05/04 22:55:00 UTC
is this a usecase for commons vfs?
Hello,
in my search for a solution for my problem I came across commons vfs.
After reading the wiki and some of the docs I'm not sure whether I can
(rather not yet, but maybe in the near future?) achieve my task with
commons vfs, hence I'm searching for enlightment.
We have an internal messaging system insite, where users can send each
other mail-like messages. They are stored on disk.
For each user we are storing a mailbox object (serialized java object
with the meta data) and plain text file for each message. There are
several hundreds of messages in each users mailbox, and millions of
users. The approach shows good io performance, but has two weaknesses,
1) we are wasting a lot of inodes with small files
2) backup times are increasing since its hard to backup a lot of small
files (snapshoting our NAS is not a problem, but creating a backup on
FS base is).
What we are looking for is a block-write fs in java, something similar
to what malloc does with memory. We want to write only files of the FS
block size to disk, but write our plain text files at defined
positions in the block files. And of course we want it to happen by
magic :-) By the end of the day we want to have something like few
4096 bytes large files on the fs with 10 496 bytes long messages in
it.
So at first it may sound like a combination of ram:// VFS resources
and a synch mechanism to dump the ram:// resources to files. Is there
something similar already available in/with VFS? If not, it it
realistic for us to implement it on top of existing VFS functionality?
thanks in advance
regards
Leon
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: is this a usecase for commons vfs?
Posted by ji...@gmail.com.
Hello Leon,
On Mon, May 4, 2009 at 4:55 PM, Leon Rosenberg
<ro...@googlemail.com> wrote:
>
> magic :-) By the end of the day we want to have something like few
> 4096 bytes large files on the fs with 10 496 bytes long messages in
> it.
>
Just a very quick point: 4096 != 10 * 496 :-)
-jesse
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: is this a usecase for commons vfs?
Posted by Mark Fortner <ph...@gmail.com>.
NIO has support for writing blocks of data to files. You have complete
byte-level control over how the data is written to the file. These examples
may give you some ideas about specific NIO features that you can apply to
solve your problem: http://www.exampledepot.com/egs/java.nio/pkg.html
Hope this helps,
Mark
On Tue, May 5, 2009 at 6:02 AM, Leon Rosenberg <
rosenberg.leon@googlemail.com> wrote:
> Thanks for the reply, but how does NIO plays into it?
>
> On Mon, May 4, 2009 at 11:59 PM, Gary Gregory
> <GG...@seagullsoftware.com> wrote:
> > Hi Leon,
> >
> > Is java.nio appropriate for your application?
> >
> > Gary
> >
> >> -----Original Message-----
> >> From: Leon Rosenberg [mailto:rosenberg.leon@googlemail.com]
> >> Sent: Monday, May 04, 2009 1:55 PM
> >> To: user@commons.apache.org
> >> Subject: is this a usecase for commons vfs?
> >>
> >> Hello,
> >>
> >> in my search for a solution for my problem I came across commons vfs.
> >> After reading the wiki and some of the docs I'm not sure whether I can
> >> (rather not yet, but maybe in the near future?) achieve my task with
> >> commons vfs, hence I'm searching for enlightment.
> >> We have an internal messaging system insite, where users can send each
> >> other mail-like messages. They are stored on disk.
> >> For each user we are storing a mailbox object (serialized java object
> >> with the meta data) and plain text file for each message. There are
> >> several hundreds of messages in each users mailbox, and millions of
> >> users. The approach shows good io performance, but has two weaknesses,
> >> 1) we are wasting a lot of inodes with small files
> >> 2) backup times are increasing since its hard to backup a lot of small
> >> files (snapshoting our NAS is not a problem, but creating a backup on
> >> FS base is).
> >>
> >> What we are looking for is a block-write fs in java, something similar
> >> to what malloc does with memory. We want to write only files of the FS
> >> block size to disk, but write our plain text files at defined
> >> positions in the block files. And of course we want it to happen by
> >> magic :-) By the end of the day we want to have something like few
> >> 4096 bytes large files on the fs with 10 496 bytes long messages in
> >> it.
> >>
> >> So at first it may sound like a combination of ram:// VFS resources
> >> and a synch mechanism to dump the ram:// resources to files. Is there
> >> something similar already available in/with VFS? If not, it it
> >> realistic for us to implement it on top of existing VFS functionality?
> >>
> >> thanks in advance
> >> regards
> >> Leon
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> >> For additional commands, e-mail: user-help@commons.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> > For additional commands, e-mail: user-help@commons.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
>
--
Mark Fortner
blog: http://feeds.feedburner.com/jroller/ideafactory
Re: is this a usecase for commons vfs?
Posted by Leon Rosenberg <ro...@googlemail.com>.
Thanks for the reply, but how does NIO plays into it?
On Mon, May 4, 2009 at 11:59 PM, Gary Gregory
<GG...@seagullsoftware.com> wrote:
> Hi Leon,
>
> Is java.nio appropriate for your application?
>
> Gary
>
>> -----Original Message-----
>> From: Leon Rosenberg [mailto:rosenberg.leon@googlemail.com]
>> Sent: Monday, May 04, 2009 1:55 PM
>> To: user@commons.apache.org
>> Subject: is this a usecase for commons vfs?
>>
>> Hello,
>>
>> in my search for a solution for my problem I came across commons vfs.
>> After reading the wiki and some of the docs I'm not sure whether I can
>> (rather not yet, but maybe in the near future?) achieve my task with
>> commons vfs, hence I'm searching for enlightment.
>> We have an internal messaging system insite, where users can send each
>> other mail-like messages. They are stored on disk.
>> For each user we are storing a mailbox object (serialized java object
>> with the meta data) and plain text file for each message. There are
>> several hundreds of messages in each users mailbox, and millions of
>> users. The approach shows good io performance, but has two weaknesses,
>> 1) we are wasting a lot of inodes with small files
>> 2) backup times are increasing since its hard to backup a lot of small
>> files (snapshoting our NAS is not a problem, but creating a backup on
>> FS base is).
>>
>> What we are looking for is a block-write fs in java, something similar
>> to what malloc does with memory. We want to write only files of the FS
>> block size to disk, but write our plain text files at defined
>> positions in the block files. And of course we want it to happen by
>> magic :-) By the end of the day we want to have something like few
>> 4096 bytes large files on the fs with 10 496 bytes long messages in
>> it.
>>
>> So at first it may sound like a combination of ram:// VFS resources
>> and a synch mechanism to dump the ram:// resources to files. Is there
>> something similar already available in/with VFS? If not, it it
>> realistic for us to implement it on top of existing VFS functionality?
>>
>> thanks in advance
>> regards
>> Leon
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>> For additional commands, e-mail: user-help@commons.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
RE: is this a usecase for commons vfs?
Posted by Gary Gregory <GG...@seagullsoftware.com>.
Hi Leon,
Is java.nio appropriate for your application?
Gary
> -----Original Message-----
> From: Leon Rosenberg [mailto:rosenberg.leon@googlemail.com]
> Sent: Monday, May 04, 2009 1:55 PM
> To: user@commons.apache.org
> Subject: is this a usecase for commons vfs?
>
> Hello,
>
> in my search for a solution for my problem I came across commons vfs.
> After reading the wiki and some of the docs I'm not sure whether I can
> (rather not yet, but maybe in the near future?) achieve my task with
> commons vfs, hence I'm searching for enlightment.
> We have an internal messaging system insite, where users can send each
> other mail-like messages. They are stored on disk.
> For each user we are storing a mailbox object (serialized java object
> with the meta data) and plain text file for each message. There are
> several hundreds of messages in each users mailbox, and millions of
> users. The approach shows good io performance, but has two weaknesses,
> 1) we are wasting a lot of inodes with small files
> 2) backup times are increasing since its hard to backup a lot of small
> files (snapshoting our NAS is not a problem, but creating a backup on
> FS base is).
>
> What we are looking for is a block-write fs in java, something similar
> to what malloc does with memory. We want to write only files of the FS
> block size to disk, but write our plain text files at defined
> positions in the block files. And of course we want it to happen by
> magic :-) By the end of the day we want to have something like few
> 4096 bytes large files on the fs with 10 496 bytes long messages in
> it.
>
> So at first it may sound like a combination of ram:// VFS resources
> and a synch mechanism to dump the ram:// resources to files. Is there
> something similar already available in/with VFS? If not, it it
> realistic for us to implement it on top of existing VFS functionality?
>
> thanks in advance
> regards
> Leon
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org