You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jeff Zhang <zj...@gmail.com> on 2009/09/29 07:37:46 UTC

Does hadoop 0.18.3 support thrift ?

Hi all,


My hadoop cluster is using hadoop 0.18.3. And I'd  like to using python to
access hdfs. But I did not found thrift in hadoop 0.18.3.

Anybody know does hadoop 0.18.3 support thrift? If not, is there any other
method to access hdfs using python ?


Thank you in advance.

Jeff zhang

Re: Does hadoop 0.18.3 support thrift ?

Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, Sep 29, 2009 at 11:20 PM, Todd Lipcon <to...@cloudera.com> wrote:
> Ah, right, it was introduced in 0.19.
>
> It shouldn't be too tough to compile against 0.18 - the API changes were
> minimal and it's an external client so internal changes shouldn't have any
> effect.
>
> I'd recommend pulling the source from 0.20, moving it into the contrib
> directory, and hacking away.
>
> If you aren't comfortable in Java and with build.xml, though, you're
> probably out of luck - it's unlikely to work without a bit of coercion.
>
> -Todd
>
>
> On Tue, Sep 29, 2009 at 8:11 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
>> But I do not found the thrift in hadoop 0.18.3.
>> Can I use other version of hadoop's thrift for my cluster which is built
>> upon hadoop 0.18.3
>>
>>
>> On Wed, Sep 30, 2009 at 1:56 AM, Todd Lipcon <to...@cloudera.com> wrote:
>>
>> > Hi Jeff,
>> >
>> > The contrib/thriftfs module is probably what you're looking for.
>> >
>> > I don't know of a lot of people using it, but it does provide a
>> proxy-style
>> > thrift access to HDFS.
>> >
>> > Be aware that performance will not be great since it introduces several
>> > more
>> > copies in the pipeline, and Thrift was never designed for transferring
>> > large
>> > amounts of data.
>> >
>> > -Todd
>> >
>> > On Mon, Sep 28, 2009 at 10:37 PM, Jeff Zhang <zj...@gmail.com> wrote:
>> >
>> > > Hi all,
>> > >
>> > >
>> > > My hadoop cluster is using hadoop 0.18.3. And I'd  like to using python
>> > to
>> > > access hdfs. But I did not found thrift in hadoop 0.18.3.
>> > >
>> > > Anybody know does hadoop 0.18.3 support thrift? If not, is there any
>> > other
>> > > method to access hdfs using python ?
>> > >
>> > >
>> > > Thank you in advance.
>> > >
>> > > Jeff zhang
>> > >
>> >
>>
>

I did this at one point. The code reference append and some other
things not in 18.0 but It does work.

Re: Does hadoop 0.18.3 support thrift ?

Posted by Todd Lipcon <to...@cloudera.com>.
Ah, right, it was introduced in 0.19.

It shouldn't be too tough to compile against 0.18 - the API changes were
minimal and it's an external client so internal changes shouldn't have any
effect.

I'd recommend pulling the source from 0.20, moving it into the contrib
directory, and hacking away.

If you aren't comfortable in Java and with build.xml, though, you're
probably out of luck - it's unlikely to work without a bit of coercion.

-Todd


On Tue, Sep 29, 2009 at 8:11 PM, Jeff Zhang <zj...@gmail.com> wrote:

> But I do not found the thrift in hadoop 0.18.3.
> Can I use other version of hadoop's thrift for my cluster which is built
> upon hadoop 0.18.3
>
>
> On Wed, Sep 30, 2009 at 1:56 AM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > Hi Jeff,
> >
> > The contrib/thriftfs module is probably what you're looking for.
> >
> > I don't know of a lot of people using it, but it does provide a
> proxy-style
> > thrift access to HDFS.
> >
> > Be aware that performance will not be great since it introduces several
> > more
> > copies in the pipeline, and Thrift was never designed for transferring
> > large
> > amounts of data.
> >
> > -Todd
> >
> > On Mon, Sep 28, 2009 at 10:37 PM, Jeff Zhang <zj...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > >
> > > My hadoop cluster is using hadoop 0.18.3. And I'd  like to using python
> > to
> > > access hdfs. But I did not found thrift in hadoop 0.18.3.
> > >
> > > Anybody know does hadoop 0.18.3 support thrift? If not, is there any
> > other
> > > method to access hdfs using python ?
> > >
> > >
> > > Thank you in advance.
> > >
> > > Jeff zhang
> > >
> >
>

Re: Does hadoop 0.18.3 support thrift ?

Posted by Jeff Zhang <zj...@gmail.com>.
But I do not found the thrift in hadoop 0.18.3.
Can I use other version of hadoop's thrift for my cluster which is built
upon hadoop 0.18.3


On Wed, Sep 30, 2009 at 1:56 AM, Todd Lipcon <to...@cloudera.com> wrote:

> Hi Jeff,
>
> The contrib/thriftfs module is probably what you're looking for.
>
> I don't know of a lot of people using it, but it does provide a proxy-style
> thrift access to HDFS.
>
> Be aware that performance will not be great since it introduces several
> more
> copies in the pipeline, and Thrift was never designed for transferring
> large
> amounts of data.
>
> -Todd
>
> On Mon, Sep 28, 2009 at 10:37 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
> > Hi all,
> >
> >
> > My hadoop cluster is using hadoop 0.18.3. And I'd  like to using python
> to
> > access hdfs. But I did not found thrift in hadoop 0.18.3.
> >
> > Anybody know does hadoop 0.18.3 support thrift? If not, is there any
> other
> > method to access hdfs using python ?
> >
> >
> > Thank you in advance.
> >
> > Jeff zhang
> >
>

Re: Does hadoop 0.18.3 support thrift ?

Posted by Todd Lipcon <to...@cloudera.com>.
Hi Jeff,

The contrib/thriftfs module is probably what you're looking for.

I don't know of a lot of people using it, but it does provide a proxy-style
thrift access to HDFS.

Be aware that performance will not be great since it introduces several more
copies in the pipeline, and Thrift was never designed for transferring large
amounts of data.

-Todd

On Mon, Sep 28, 2009 at 10:37 PM, Jeff Zhang <zj...@gmail.com> wrote:

> Hi all,
>
>
> My hadoop cluster is using hadoop 0.18.3. And I'd  like to using python to
> access hdfs. But I did not found thrift in hadoop 0.18.3.
>
> Anybody know does hadoop 0.18.3 support thrift? If not, is there any other
> method to access hdfs using python ?
>
>
> Thank you in advance.
>
> Jeff zhang
>