Re: Can I replace the pig backend with hadoop streming?

[devdoer bird <de...@gmail.com>]

Thanks.

2012/1/10 Dmitriy Ryaboy <dv...@gmail.com>

> Oh and as luck would have it, it was made for the 0.4-0.5 release.
>
> On Mon, Jan 9, 2012 at 1:13 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>
> > I had a patch that introduced shims, which included shims to make Pig
> work
> > with 19. It's woefully out of date at this point, but you can at least
> use
> > it as a starting point: https://issues.apache.org/jira/browse/PIG-924
> >
> >
> > On Mon, Jan 9, 2012 at 11:23 AM, Daniel Dai <da...@hortonworks.com>
> wrote:
> >
> >> There are some API incompatible changes between 0.19 and 0.20. You will
> >> need to change current Pig code in order to compile/run against 0.19.
> But
> >> the cost of downgrade hadoop should be much lower than write a new
> >> backend.
> >> You can check PIG-660 (in which we upgrade hadoop 18 to 20) to get some
> >> idea.
> >>
> >> Daniel
> >>
> >> On Mon, Jan 9, 2012 at 2:01 AM, devdoer bird <de...@gmail.com>
> wrote:
> >>
> >> > If possible I want to use the official pig distribution, but the
> >> company's
> >> > hadoop (from hadoop 0.19) backend has changed a lot of api from the
> >> > official one which I can't get the documentaion.
> >> >
> >> > I tried to fix the imcompatial problem   by making  pig 0.9.1  work
> with
> >> > official hadoop 0.19 ,then work with my company's private hadoop
> >> > distribution . But I don't where to begin.
> >> >
> >> >  Any help will be welcome.
> >> >
> >> > 2012/1/9 Daniel Dai <da...@hortonworks.com>
> >> >
> >> > > What error did you see when you use official Pig distribution? It's
> >> > > non-trivial to write a backend even if possible (Especially for 0.5,
> >> it
> >> > > would be hard to get help from the community)
> >> > >
> >> > > On Mon, Jan 9, 2012 at 12:16 AM, devdoer bird <de...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > So I think Streaming backend may be a good solution for this
> >> situation.
> >> > > >
> >> > > > 2012/1/9 devdoer bird <de...@gmail.com>
> >> > > >
> >> > > > > The reason I decide use Streaming as backend is that  In my
> >> company
> >> > the
> >> > > > > hadoop has been modified  so it might not be compatible with the
> >> > > official
> >> > > > > hadoop distribution. And I can't make  the pig run on our
> private
> >> > > hadoop
> >> > > > > distribution.
> >> > > > >
> >> > > > >
> >> > > > > 2012/1/9 Daniel Dai <da...@hortonworks.com>
> >> > > > >
> >> > > > >> Pig do have a abstraction layer in execution engine. But that
> is
> >> > > mostly
> >> > > > a
> >> > > > >> legacy of early versions. In recent development, we never keep
> >> > > platform
> >> > > > >> neutral in mind so I don't know how reliable this interface is.
> >> Can
> >> > > you
> >> > > > >> elaborate your idea so we may find a better solution?
> >> > > > >>
> >> > > > >> Daniel
> >> > > > >>
> >> > > > >> On Sun, Jan 8, 2012 at 7:50 PM, devdoer bird <
> devdoer2@gmail.com
> >> >
> >> > > > wrote:
> >> > > > >>
> >> > > > >> > HI:
> >> > > > >> >
> >> > > > >> > I want to implement a new pig backend  . Can I replache the
> >> hadoop
> >> > > > >> backend
> >> > > > >> > with a hadoop--streaming only backend?
> >> > > > >> >
> >> > > > >> > I decide to use streaming to implement backend.storage and
> >> backend
> >> > > > >> > .executionengine interface , but I want to know   whether
> it's
> >> a
> >> > > right
> >> > > > >> way
> >> > > > >> > to do so.
> >> > > > >> >
> >> > > > >> > Thanks.
> >> > > > >> >
> >> > > > >>
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>