You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Steve Loughran <st...@hortonworks.com> on 2013/06/03 16:12:53 UTC
HADOOP-8545 OpenStack Swift patch ready for commit -reviewers needed
Hi,
We've got the HADOOP-8545<https://issues.apache.org/jira/browse/HADOOP-8545>patch
stable enough that I'd like it checked in and in the 2.1 beta so that
we can see what problems occur in the field.
The patch adds a new module, hadoop-tools/hadoop-openstack, so it doesn't
break anything in hadoop-common/hdfs/yarn, and it includes more
method-by-method tests than any of the non-HDFS filesystem clients have,
including basic scale tests and the corner-case problems that we're going
to have to talk about in terms of FS API spec and tests. In github, there
are some very basic scale tests running Pig jobs against a configurable
amount of data, showing that we can run pig against GB of data.
We've run these tests against rackspace US, rackspace US and hpcloud -the
public Swift services -as well as private deployments of the latest version
of OpenStack. This makes us confident that we've got authentication right,
and as we get consistent outcomes from all clusters, we aren't relying some
cluster-/provider- specific detail.
So: please can people review it! The site documentation (did I mention
we've got that?) lists how to connect to a swift service, and how to test
against them. If anyone does want to run the test themselves, try the docs
and then escalate to us if there are any problems.
Steve Loughran, Dmitry Mezhenskiy (Mirantis) & David Dobbins (Rackspace)
Re: HADOOP-8545 OpenStack Swift patch ready for commit -reviewers needed
Posted by Steve Loughran <st...@hortonworks.com>.
thanks
On 3 June 2013 17:20, Suresh Srinivas <su...@hortonworks.com> wrote:
> Steve, I will review this. Might need a couple of days though.
>
>
> On Mon, Jun 3, 2013 at 7:12 AM, Steve Loughran <stevel@hortonworks.com
> >wrote:
>
> > Hi,
> >
> > We've got the HADOOP-8545<
> > https://issues.apache.org/jira/browse/HADOOP-8545>patch
> > stable enough that I'd like it checked in and in the 2.1 beta so that
> > we can see what problems occur in the field.
> >
> > The patch adds a new module, hadoop-tools/hadoop-openstack, so it doesn't
> > break anything in hadoop-common/hdfs/yarn, and it includes more
> > method-by-method tests than any of the non-HDFS filesystem clients have,
> > including basic scale tests and the corner-case problems that we're going
> > to have to talk about in terms of FS API spec and tests. In github, there
> > are some very basic scale tests running Pig jobs against a configurable
> > amount of data, showing that we can run pig against GB of data.
> >
> > We've run these tests against rackspace US, rackspace US and hpcloud -the
> > public Swift services -as well as private deployments of the latest
> version
> > of OpenStack. This makes us confident that we've got authentication
> right,
> > and as we get consistent outcomes from all clusters, we aren't relying
> some
> > cluster-/provider- specific detail.
> >
> > So: please can people review it! The site documentation (did I mention
> > we've got that?) lists how to connect to a swift service, and how to test
> > against them. If anyone does want to run the test themselves, try the
> docs
> > and then escalate to us if there are any problems.
> >
> >
> > Steve Loughran, Dmitry Mezhenskiy (Mirantis) & David Dobbins (Rackspace)
> >
>
>
>
> --
> http://hortonworks.com/download/
>
Re: HADOOP-8545 OpenStack Swift patch ready for commit -reviewers needed
Posted by Suresh Srinivas <su...@hortonworks.com>.
Steve, I will review this. Might need a couple of days though.
On Mon, Jun 3, 2013 at 7:12 AM, Steve Loughran <st...@hortonworks.com>wrote:
> Hi,
>
> We've got the HADOOP-8545<
> https://issues.apache.org/jira/browse/HADOOP-8545>patch
> stable enough that I'd like it checked in and in the 2.1 beta so that
> we can see what problems occur in the field.
>
> The patch adds a new module, hadoop-tools/hadoop-openstack, so it doesn't
> break anything in hadoop-common/hdfs/yarn, and it includes more
> method-by-method tests than any of the non-HDFS filesystem clients have,
> including basic scale tests and the corner-case problems that we're going
> to have to talk about in terms of FS API spec and tests. In github, there
> are some very basic scale tests running Pig jobs against a configurable
> amount of data, showing that we can run pig against GB of data.
>
> We've run these tests against rackspace US, rackspace US and hpcloud -the
> public Swift services -as well as private deployments of the latest version
> of OpenStack. This makes us confident that we've got authentication right,
> and as we get consistent outcomes from all clusters, we aren't relying some
> cluster-/provider- specific detail.
>
> So: please can people review it! The site documentation (did I mention
> we've got that?) lists how to connect to a swift service, and how to test
> against them. If anyone does want to run the test themselves, try the docs
> and then escalate to us if there are any problems.
>
>
> Steve Loughran, Dmitry Mezhenskiy (Mirantis) & David Dobbins (Rackspace)
>
--
http://hortonworks.com/download/