You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Mark Diggory <md...@gmail.com> on 2009/12/21 05:18:14 UTC
Amazon EC2 EBS Volume for Maven central repository
Cheers,
After experimenting a bit with EC2, It seems it would be ideal if
there were an EBS volume that had all the Maven central repository
within it. Does anyone know of such a thing?
thanks,
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Amazon EC2 EBS Volume for Maven central repository
Posted by Carlos Sanchez <ca...@apache.org>.
the problem for a public dataset it's that AFAIK they are static while
the repo keeps changing
On Mon, Dec 21, 2009 at 12:55 PM, Mark Diggory <md...@gmail.com> wrote:
> FYI
>
> Initiating a thread here to see its effect.
>
> http://developer.amazonwebservices.com/connect/thread.jspa?messageID=158762𦰪
>
> Cheers,
> Mark
>
> On Mon, Dec 21, 2009 at 10:19 AM, Carlos Sanchez <ca...@apache.org> wrote:
>
>> you can set s3 buckets where "requester pays"
>>
>>
>> http://developer.amazonwebservices.com/connect/message.jspa?messageID=123715
>>
>> On Mon, Dec 21, 2009 at 9:30 AM, Brian Fox <br...@infinity.nu> wrote:
>> > On Mon, Dec 21, 2009 at 12:09 AM, Mark Diggory <md...@gmail.com>
>> wrote:
>> >> I imagine there have to be a number of projects/companies out there
>> >> using Maven artifacts and incurring bandwidth costs to build systems.
>> >> Atlassian seems to be recommending the practice to its Bamboo users...
>> >
>> >
>> > They should just have a repo manager running in their ec2 grid anyway
>> > to reduce bandwidth.
>> >
>> > We looked into putting central into s3 but then you loose all control
>> > over the access and we would get killed by the scrapers constantly
>> > downloading everything because it's "free".
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Amazon EC2 EBS Volume for Maven central repository
Posted by Mark Diggory <md...@gmail.com>.
FYI
Initiating a thread here to see its effect.
http://developer.amazonwebservices.com/connect/thread.jspa?messageID=158762𦰪
Cheers,
Mark
On Mon, Dec 21, 2009 at 10:19 AM, Carlos Sanchez <ca...@apache.org> wrote:
> you can set s3 buckets where "requester pays"
>
>
> http://developer.amazonwebservices.com/connect/message.jspa?messageID=123715
>
> On Mon, Dec 21, 2009 at 9:30 AM, Brian Fox <br...@infinity.nu> wrote:
> > On Mon, Dec 21, 2009 at 12:09 AM, Mark Diggory <md...@gmail.com>
> wrote:
> >> I imagine there have to be a number of projects/companies out there
> >> using Maven artifacts and incurring bandwidth costs to build systems.
> >> Atlassian seems to be recommending the practice to its Bamboo users...
> >
> >
> > They should just have a repo manager running in their ec2 grid anyway
> > to reduce bandwidth.
> >
> > We looked into putting central into s3 but then you loose all control
> > over the access and we would get killed by the scrapers constantly
> > downloading everything because it's "free".
>
Re: Amazon EC2 EBS Volume for Maven central repository
Posted by Carlos Sanchez <ca...@apache.org>.
you can set s3 buckets where "requester pays"
http://developer.amazonwebservices.com/connect/message.jspa?messageID=123715
On Mon, Dec 21, 2009 at 9:30 AM, Brian Fox <br...@infinity.nu> wrote:
> On Mon, Dec 21, 2009 at 12:09 AM, Mark Diggory <md...@gmail.com> wrote:
>> I imagine there have to be a number of projects/companies out there
>> using Maven artifacts and incurring bandwidth costs to build systems.
>> Atlassian seems to be recommending the practice to its Bamboo users...
>
>
> They should just have a repo manager running in their ec2 grid anyway
> to reduce bandwidth.
>
> We looked into putting central into s3 but then you loose all control
> over the access and we would get killed by the scrapers constantly
> downloading everything because it's "free".
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Amazon EC2 EBS Volume for Maven central repository
Posted by Brian Fox <br...@infinity.nu>.
On Mon, Dec 21, 2009 at 12:09 AM, Mark Diggory <md...@gmail.com> wrote:
> I imagine there have to be a number of projects/companies out there
> using Maven artifacts and incurring bandwidth costs to build systems.
> Atlassian seems to be recommending the practice to its Bamboo users...
They should just have a repo manager running in their ec2 grid anyway
to reduce bandwidth.
We looked into putting central into s3 but then you loose all control
over the access and we would get killed by the scrapers constantly
downloading everything because it's "free".
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Amazon EC2 EBS Volume for Maven central repository
Posted by Mark Diggory <md...@gmail.com>.
Ok, so, from the EBS web page...we can speculate...
As an example, a medium sized website database might be 100 GB in size and
> expect to average 100 I/Os per second over the course of a month. This would
> translate to $10 per month in storage costs (100 GB x $0.10/month), and
> approximately $26 per month in request costs (~2.6 million seconds/month x
> 100 I/O per second * $0.10 per million I/O).
>
We might seek out the Freebase folks and inquire for the what their costs
are like to provide an EBS dataset.
Mark
On Mon, Dec 21, 2009 at 12:43 AM, Wayne Fay <wa...@gmail.com> wrote:
>> What's the current storage requirements for the central repository at
this time?
>>
>
> Per Jarvana, Central is around 100gb, as of mid October 2009:
> http://www.jarvana.com/jarvana/info/repository_statistics
>
> Wayne
>
> On Sun, Dec 20, 2009 at 9:09 PM, Mark Diggory <md...@gmail.com> wrote:
> I imagine there have to be a number of projects/companies out there
> using Maven artifacts and incurring bandwidth costs to build systems.
> Atlassian seems to be recommending the practice to its Bamboo users...
>
>
> http://confluence.atlassian.com/display/BAMBOO/Populating+your+EBS+volume#PopulatingyourEBSvolume-UploadingMaven2repositorydata
>
> I know its not much, but on large builds and continuous integration
> projects would benefit.
>
> What's the current storage requirements for the central repository at this
> time?
>
> I wonder if the maven central repo could qualify as a public dataset?
> http://aws.amazon.com/publicdatasets/
> http://developer.amazonwebservices.com/connect/forum.jspa?forumID=55
>
> Mark
>
> On Sun, Dec 20, 2009 at 8:28 PM, Carlos Sanchez <ca...@apache.org> wrote:
> > there was some talk about it during ApacheCON but there are some costs
> > associated
> >
> > - S3 storage of the EBS snapshot
> > - bandwidth to upload the repo and keep it up to date
> > - an EC2 machine to do the rsync and scheduled snapshots of the EBS
> volume
> >
> > There's an option to "sell" it so users end paying the fees
> >
> >
> > On Sun, Dec 20, 2009 at 8:18 PM, Mark Diggory <md...@gmail.com>
> wrote:
> >> Cheers,
> >>
> >> After experimenting a bit with EC2, It seems it would be ideal if
> >> there were an EBS volume that had all the Maven central repository
> >> within it. Does anyone know of such a thing?
> >>
> >> thanks,
> >> Mark
>
>
Re: Amazon EC2 EBS Volume for Maven central repository
Posted by Wayne Fay <wa...@gmail.com>.
> What's the current storage requirements for the central repository at this time?
>
Per Jarvana, Central is around 100gb, as of mid October 2009:
http://www.jarvana.com/jarvana/info/repository_statistics
Wayne
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Amazon EC2 EBS Volume for Maven central repository
Posted by Mark Diggory <md...@gmail.com>.
I imagine there have to be a number of projects/companies out there
using Maven artifacts and incurring bandwidth costs to build systems.
Atlassian seems to be recommending the practice to its Bamboo users...
http://confluence.atlassian.com/display/BAMBOO/Populating+your+EBS+volume#PopulatingyourEBSvolume-UploadingMaven2repositorydata
I know its not much, but on large builds and continuous integration
projects would benefit.
What's the current storage requirements for the central repository at this time?
I wonder if the maven central repo could qualify as a public dataset?
http://aws.amazon.com/publicdatasets/
http://developer.amazonwebservices.com/connect/forum.jspa?forumID=55
Mark
On Sun, Dec 20, 2009 at 8:28 PM, Carlos Sanchez <ca...@apache.org> wrote:
> there was some talk about it during ApacheCON but there are some costs
> associated
>
> - S3 storage of the EBS snapshot
> - bandwidth to upload the repo and keep it up to date
> - an EC2 machine to do the rsync and scheduled snapshots of the EBS volume
>
> There's an option to "sell" it so users end paying the fees
>
>
> On Sun, Dec 20, 2009 at 8:18 PM, Mark Diggory <md...@gmail.com> wrote:
>> Cheers,
>>
>> After experimenting a bit with EC2, It seems it would be ideal if
>> there were an EBS volume that had all the Maven central repository
>> within it. Does anyone know of such a thing?
>>
>> thanks,
>> Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Amazon EC2 EBS Volume for Maven central repository
Posted by Carlos Sanchez <ca...@apache.org>.
there was some talk about it during ApacheCON but there are some costs
associated
- S3 storage of the EBS snapshot
- bandwidth to upload the repo and keep it up to date
- an EC2 machine to do the rsync and scheduled snapshots of the EBS volume
There's an option to "sell" it so users end paying the fees
On Sun, Dec 20, 2009 at 8:18 PM, Mark Diggory <md...@gmail.com> wrote:
> Cheers,
>
> After experimenting a bit with EC2, It seems it would be ideal if
> there were an EBS volume that had all the Maven central repository
> within it. Does anyone know of such a thing?
>
> thanks,
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org