You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by Martin Perez <mp...@gmail.com> on 2007/06/04 14:38:34 UTC
S3 persistence manager
Hi guys,
Is there anyone developing something like a Amazon S3 persistence manager?
If not, it seems to me like a good candidate for a new project ( it is too
late for SOC ), or perhaps it is an insane idea?
Cheers,
Martin
Re: S3 persistence manager
Posted by Jorge Ferrer <jo...@gmail.com>.
Hi Martin,
Thanks for your answer. I'll take a look at the filesystem
implementation from Hadoop.
Also, I've found jets3t [1] which seem to simplify development with
Java on top of S3. We'll probably use it to build a less-featured
alternative to Jackrabbit for now and will think about implementing a
PM with it later on.
Regards,
Jorge
[1] https://jets3t.dev.java.net/
On 7/2/07, Martin Perez <mp...@gmail.com> wrote:
> I don't think anyone is currently looking at this. It was only a random idea
> that I dropped at the forum.
>
> However, seeing that another Apache project has already a S3 filesystem
> built in perhaps it would be interesting to look at it (
> http://wiki.apache.org/lucene-hadoop/AmazonS3)
>
> Cheers,
> Martin
>
> On 7/2/07, Jorge Ferrer <jo...@gmail.com> wrote:
> >
> > Hi Jukka, Martin,
> >
> > Has there been any progress on this? I'm also interested in this and
> > would like to help.
> >
> > Related to Jerome's suggestions, do you think that the current PM
> > architecture perform well if using a FileSystem PM to a directory that
> > is actually a S3 FileSystem [1]?
> >
> > Regards,
> > Jorge
> >
> > [1]
> > http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&tstart=30
> >
> > On 6/9/07, Jerome B <je...@gmail.com> wrote:
> > >
> > >
> > > I think there are projects which access S3 as a WebDAV store, as well
> > as S3
> > > "filesystems". If there was a S3 Jackrabbit persistence manager being
> > > developed, I'd be interested as well.
> > >
> > > -- jerome
> > >
> > >
> > > Martin Perez wrote:
> > > >
> > > > Really glad to read that. Massive distributed storage propositions are
> > > > being
> > > > proven like very good solutions for massive scaling. I guess the key
> > > > challenge in this case is trying to minimize bandwidth usage as much
> > as
> > > > possible in addition to the traditional storage challenge.
> > > >
> > > > Would be really nice to test it when that global data store stuff you
> > are
> > > > working in is done. If at that time I have some web projec using
> > > > Jackrabbit
> > > > (hopefully) I would love to spend some time playing with that.
> > However, It
> > > > would be even better if someone already has experiences or is already
> > > > trying
> > > > to do it and shares his experience.
> > > >
> > > > Cheers,
> > > > Martin
> > > >
> > > > On 6/5/07, Jukka Zitting <ju...@gmail.com> wrote:
> > > >>
> > > >> Hi,
> > > >>
> > > >> On 6/4/07, Martin Perez <mp...@gmail.com> wrote:
> > > >> > Is there anyone developing something like a Amazon S3 persistence
> > > >> manager?
> > > >> >
> > > >> > If not, it seems to me like a good candidate for a new project ( it
> > is
> > > >> too
> > > >> > late for SOC ), or perhaps it is an insane idea?
> > > >>
> > > >> I haven't heard of anyone actively working on something like that,
> > but
> > > >> I agree that it would be a cool idea.
> > > >>
> > > >> Actually S3 was floating around in my mind when I wrote the NGP
> > > >> proposal (http://jackrabbit.apache.org/dev/ngp.html). S3 or another
> > > >> similar massively distributed storage mechanism would work very well
> > > >> with the NGP concepts.
> > > >>
> > > >> I'm currently working on a related more short-term idea of adding a
> > > >> global "data store" that would handle all binary properties (and
> > > >> possibly other data like item bundles as well). The data store
> > concept
> > > >> is designed to work very well with remote or distributed storage, so
> > > >> S3 would be an interesting option there as well.
> > > >>
> > > >> BR,
> > > >>
> > > >> Jukka Zitting
> > > >>
> > > >
> > > >
> > >
> > > --
> > > View this message in context:
> > http://www.nabble.com/S3-persistence-manager-tf3864934.html#a11036005
> > > Sent from the Jackrabbit - Dev mailing list archive at Nabble.com.
> > >
> > >
> >
> >
> > --
> > Jorge Ferrer Zarzuela
> > http://www.jorgeferrer.com
> > http://www.liferay.com
> >
>
--
Jorge Ferrer Zarzuela
http://www.jorgeferrer.com
http://www.liferay.com
Re: S3 persistence manager
Posted by Martin Perez <mp...@gmail.com>.
I don't think anyone is currently looking at this. It was only a random idea
that I dropped at the forum.
However, seeing that another Apache project has already a S3 filesystem
built in perhaps it would be interesting to look at it (
http://wiki.apache.org/lucene-hadoop/AmazonS3)
Cheers,
Martin
On 7/2/07, Jorge Ferrer <jo...@gmail.com> wrote:
>
> Hi Jukka, Martin,
>
> Has there been any progress on this? I'm also interested in this and
> would like to help.
>
> Related to Jerome's suggestions, do you think that the current PM
> architecture perform well if using a FileSystem PM to a directory that
> is actually a S3 FileSystem [1]?
>
> Regards,
> Jorge
>
> [1]
> http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&tstart=30
>
> On 6/9/07, Jerome B <je...@gmail.com> wrote:
> >
> >
> > I think there are projects which access S3 as a WebDAV store, as well
> as S3
> > "filesystems". If there was a S3 Jackrabbit persistence manager being
> > developed, I'd be interested as well.
> >
> > -- jerome
> >
> >
> > Martin Perez wrote:
> > >
> > > Really glad to read that. Massive distributed storage propositions are
> > > being
> > > proven like very good solutions for massive scaling. I guess the key
> > > challenge in this case is trying to minimize bandwidth usage as much
> as
> > > possible in addition to the traditional storage challenge.
> > >
> > > Would be really nice to test it when that global data store stuff you
> are
> > > working in is done. If at that time I have some web projec using
> > > Jackrabbit
> > > (hopefully) I would love to spend some time playing with that.
> However, It
> > > would be even better if someone already has experiences or is already
> > > trying
> > > to do it and shares his experience.
> > >
> > > Cheers,
> > > Martin
> > >
> > > On 6/5/07, Jukka Zitting <ju...@gmail.com> wrote:
> > >>
> > >> Hi,
> > >>
> > >> On 6/4/07, Martin Perez <mp...@gmail.com> wrote:
> > >> > Is there anyone developing something like a Amazon S3 persistence
> > >> manager?
> > >> >
> > >> > If not, it seems to me like a good candidate for a new project ( it
> is
> > >> too
> > >> > late for SOC ), or perhaps it is an insane idea?
> > >>
> > >> I haven't heard of anyone actively working on something like that,
> but
> > >> I agree that it would be a cool idea.
> > >>
> > >> Actually S3 was floating around in my mind when I wrote the NGP
> > >> proposal (http://jackrabbit.apache.org/dev/ngp.html). S3 or another
> > >> similar massively distributed storage mechanism would work very well
> > >> with the NGP concepts.
> > >>
> > >> I'm currently working on a related more short-term idea of adding a
> > >> global "data store" that would handle all binary properties (and
> > >> possibly other data like item bundles as well). The data store
> concept
> > >> is designed to work very well with remote or distributed storage, so
> > >> S3 would be an interesting option there as well.
> > >>
> > >> BR,
> > >>
> > >> Jukka Zitting
> > >>
> > >
> > >
> >
> > --
> > View this message in context:
> http://www.nabble.com/S3-persistence-manager-tf3864934.html#a11036005
> > Sent from the Jackrabbit - Dev mailing list archive at Nabble.com.
> >
> >
>
>
> --
> Jorge Ferrer Zarzuela
> http://www.jorgeferrer.com
> http://www.liferay.com
>
Re: S3 persistence manager
Posted by Jorge Ferrer <jo...@gmail.com>.
Hi Jukka, Martin,
Has there been any progress on this? I'm also interested in this and
would like to help.
Related to Jerome's suggestions, do you think that the current PM
architecture perform well if using a FileSystem PM to a directory that
is actually a S3 FileSystem [1]?
Regards,
Jorge
[1] http://developer.amazonwebservices.com/connect/thread.jspa?threadID=10271&tstart=30
On 6/9/07, Jerome B <je...@gmail.com> wrote:
>
>
> I think there are projects which access S3 as a WebDAV store, as well as S3
> "filesystems". If there was a S3 Jackrabbit persistence manager being
> developed, I'd be interested as well.
>
> -- jerome
>
>
> Martin Perez wrote:
> >
> > Really glad to read that. Massive distributed storage propositions are
> > being
> > proven like very good solutions for massive scaling. I guess the key
> > challenge in this case is trying to minimize bandwidth usage as much as
> > possible in addition to the traditional storage challenge.
> >
> > Would be really nice to test it when that global data store stuff you are
> > working in is done. If at that time I have some web projec using
> > Jackrabbit
> > (hopefully) I would love to spend some time playing with that. However, It
> > would be even better if someone already has experiences or is already
> > trying
> > to do it and shares his experience.
> >
> > Cheers,
> > Martin
> >
> > On 6/5/07, Jukka Zitting <ju...@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> On 6/4/07, Martin Perez <mp...@gmail.com> wrote:
> >> > Is there anyone developing something like a Amazon S3 persistence
> >> manager?
> >> >
> >> > If not, it seems to me like a good candidate for a new project ( it is
> >> too
> >> > late for SOC ), or perhaps it is an insane idea?
> >>
> >> I haven't heard of anyone actively working on something like that, but
> >> I agree that it would be a cool idea.
> >>
> >> Actually S3 was floating around in my mind when I wrote the NGP
> >> proposal (http://jackrabbit.apache.org/dev/ngp.html). S3 or another
> >> similar massively distributed storage mechanism would work very well
> >> with the NGP concepts.
> >>
> >> I'm currently working on a related more short-term idea of adding a
> >> global "data store" that would handle all binary properties (and
> >> possibly other data like item bundles as well). The data store concept
> >> is designed to work very well with remote or distributed storage, so
> >> S3 would be an interesting option there as well.
> >>
> >> BR,
> >>
> >> Jukka Zitting
> >>
> >
> >
>
> --
> View this message in context: http://www.nabble.com/S3-persistence-manager-tf3864934.html#a11036005
> Sent from the Jackrabbit - Dev mailing list archive at Nabble.com.
>
>
--
Jorge Ferrer Zarzuela
http://www.jorgeferrer.com
http://www.liferay.com
Re: S3 persistence manager
Posted by Jerome B <je...@gmail.com>.
I think there are projects which access S3 as a WebDAV store, as well as S3
"filesystems". If there was a S3 Jackrabbit persistence manager being
developed, I'd be interested as well.
-- jerome
Martin Perez wrote:
>
> Really glad to read that. Massive distributed storage propositions are
> being
> proven like very good solutions for massive scaling. I guess the key
> challenge in this case is trying to minimize bandwidth usage as much as
> possible in addition to the traditional storage challenge.
>
> Would be really nice to test it when that global data store stuff you are
> working in is done. If at that time I have some web projec using
> Jackrabbit
> (hopefully) I would love to spend some time playing with that. However, It
> would be even better if someone already has experiences or is already
> trying
> to do it and shares his experience.
>
> Cheers,
> Martin
>
> On 6/5/07, Jukka Zitting <ju...@gmail.com> wrote:
>>
>> Hi,
>>
>> On 6/4/07, Martin Perez <mp...@gmail.com> wrote:
>> > Is there anyone developing something like a Amazon S3 persistence
>> manager?
>> >
>> > If not, it seems to me like a good candidate for a new project ( it is
>> too
>> > late for SOC ), or perhaps it is an insane idea?
>>
>> I haven't heard of anyone actively working on something like that, but
>> I agree that it would be a cool idea.
>>
>> Actually S3 was floating around in my mind when I wrote the NGP
>> proposal (http://jackrabbit.apache.org/dev/ngp.html). S3 or another
>> similar massively distributed storage mechanism would work very well
>> with the NGP concepts.
>>
>> I'm currently working on a related more short-term idea of adding a
>> global "data store" that would handle all binary properties (and
>> possibly other data like item bundles as well). The data store concept
>> is designed to work very well with remote or distributed storage, so
>> S3 would be an interesting option there as well.
>>
>> BR,
>>
>> Jukka Zitting
>>
>
>
--
View this message in context: http://www.nabble.com/S3-persistence-manager-tf3864934.html#a11036005
Sent from the Jackrabbit - Dev mailing list archive at Nabble.com.
Re: S3 persistence manager
Posted by Martin Perez <mp...@gmail.com>.
Really glad to read that. Massive distributed storage propositions are being
proven like very good solutions for massive scaling. I guess the key
challenge in this case is trying to minimize bandwidth usage as much as
possible in addition to the traditional storage challenge.
Would be really nice to test it when that global data store stuff you are
working in is done. If at that time I have some web projec using Jackrabbit
(hopefully) I would love to spend some time playing with that. However, It
would be even better if someone already has experiences or is already trying
to do it and shares his experience.
Cheers,
Martin
On 6/5/07, Jukka Zitting <ju...@gmail.com> wrote:
>
> Hi,
>
> On 6/4/07, Martin Perez <mp...@gmail.com> wrote:
> > Is there anyone developing something like a Amazon S3 persistence
> manager?
> >
> > If not, it seems to me like a good candidate for a new project ( it is
> too
> > late for SOC ), or perhaps it is an insane idea?
>
> I haven't heard of anyone actively working on something like that, but
> I agree that it would be a cool idea.
>
> Actually S3 was floating around in my mind when I wrote the NGP
> proposal (http://jackrabbit.apache.org/dev/ngp.html). S3 or another
> similar massively distributed storage mechanism would work very well
> with the NGP concepts.
>
> I'm currently working on a related more short-term idea of adding a
> global "data store" that would handle all binary properties (and
> possibly other data like item bundles as well). The data store concept
> is designed to work very well with remote or distributed storage, so
> S3 would be an interesting option there as well.
>
> BR,
>
> Jukka Zitting
>
Re: S3 persistence manager
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On 6/4/07, Martin Perez <mp...@gmail.com> wrote:
> Is there anyone developing something like a Amazon S3 persistence manager?
>
> If not, it seems to me like a good candidate for a new project ( it is too
> late for SOC ), or perhaps it is an insane idea?
I haven't heard of anyone actively working on something like that, but
I agree that it would be a cool idea.
Actually S3 was floating around in my mind when I wrote the NGP
proposal (http://jackrabbit.apache.org/dev/ngp.html). S3 or another
similar massively distributed storage mechanism would work very well
with the NGP concepts.
I'm currently working on a related more short-term idea of adding a
global "data store" that would handle all binary properties (and
possibly other data like item bundles as well). The data store concept
is designed to work very well with remote or distributed storage, so
S3 would be an interesting option there as well.
BR,
Jukka Zitting
S3 persistence manager
Posted by Martin Perez <mp...@gmail.com>.
Hi guys,
Is there anyone developing something like a Amazon S3 persistence manager?
If not, it seems to me like a good candidate for a new project ( it is too
late for SOC ), or perhaps it is an insane idea?
Cheers,
Martin