You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Serega Sheypak <se...@gmail.com> on 2013/12/15 15:07:57 UTC
Unoverride load in PigUint
Hi!
By default PigUnit does override LOAD statements
Is there any possiblity to void this?
I'm using AvroStorage because of evolving schemas and I would like to write
in my tests:
--load test dataset
in = LOAD '$pathToIn' using AvroStorage();
--project the fields I need
inProjected = FOREACH in GENERATE f1, f2;
Right now I can't do this. PigUnit requires explicit field projection on
LOAD statement.
Re: Unoverride load in PigUint
Posted by Ruslan Al-Fakikh <me...@gmail.com>.
I did
On Sat, Dec 21, 2013 at 9:44 PM, Serega Sheypak <se...@gmail.com>wrote:
> https://issues.apache.org/jira/browse/PIG-3638
> "like" it :)
>
>
> 2013/12/21 Ruslan Al-Fakikh <me...@gmail.com>
>
> > It seems to be a heavy PigUnit limitation. Maybe you can open a jira for
> > this?:)
> >
> > Thanks,
> > Ruslan
> >
> >
> > On Sat, Dec 21, 2013 at 2:01 AM, Serega Sheypak <
> serega.sheypak@gmail.com
> > >wrote:
> >
> > > No :)
> > > I did extend PigUnit to work with several input relations (i mean 1..N
> > load
> > > statements), but I didn't find a way to use "native" loader/storage.
> > > Looks like that the only solution is to create wrapper: data-driven
> > tester
> > > which feeds script to local pig server and verifies output.
> > > We did in Megafon. I tried to use "recommended approach" - pig unit for
> > my
> > > own purposes.
> > > Fail.
> > >
> > >
> > > 2013/12/21 Ruslan Al-Fakikh <me...@gmail.com>
> > >
> > > > Hi Serega!
> > > >
> > > > Have you resolved the issue? I am going to encounter the same
> problem,
> > > but
> > > > I don't know a solution.
> > > >
> > > > Thanks
> > > >
> > > >
> > > > On Sun, Dec 15, 2013 at 6:07 PM, Serega Sheypak <
> > > serega.sheypak@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi!
> > > > > By default PigUnit does override LOAD statements
> > > > > Is there any possiblity to void this?
> > > > > I'm using AvroStorage because of evolving schemas and I would like
> to
> > > > write
> > > > > in my tests:
> > > > >
> > > > > --load test dataset
> > > > > in = LOAD '$pathToIn' using AvroStorage();
> > > > >
> > > > > --project the fields I need
> > > > > inProjected = FOREACH in GENERATE f1, f2;
> > > > >
> > > > > Right now I can't do this. PigUnit requires explicit field
> projection
> > > on
> > > > > LOAD statement.
> > > > >
> > > >
> > >
> >
>
Re: Unoverride load in PigUint
Posted by Serega Sheypak <se...@gmail.com>.
https://issues.apache.org/jira/browse/PIG-3638
"like" it :)
2013/12/21 Ruslan Al-Fakikh <me...@gmail.com>
> It seems to be a heavy PigUnit limitation. Maybe you can open a jira for
> this?:)
>
> Thanks,
> Ruslan
>
>
> On Sat, Dec 21, 2013 at 2:01 AM, Serega Sheypak <serega.sheypak@gmail.com
> >wrote:
>
> > No :)
> > I did extend PigUnit to work with several input relations (i mean 1..N
> load
> > statements), but I didn't find a way to use "native" loader/storage.
> > Looks like that the only solution is to create wrapper: data-driven
> tester
> > which feeds script to local pig server and verifies output.
> > We did in Megafon. I tried to use "recommended approach" - pig unit for
> my
> > own purposes.
> > Fail.
> >
> >
> > 2013/12/21 Ruslan Al-Fakikh <me...@gmail.com>
> >
> > > Hi Serega!
> > >
> > > Have you resolved the issue? I am going to encounter the same problem,
> > but
> > > I don't know a solution.
> > >
> > > Thanks
> > >
> > >
> > > On Sun, Dec 15, 2013 at 6:07 PM, Serega Sheypak <
> > serega.sheypak@gmail.com
> > > >wrote:
> > >
> > > > Hi!
> > > > By default PigUnit does override LOAD statements
> > > > Is there any possiblity to void this?
> > > > I'm using AvroStorage because of evolving schemas and I would like to
> > > write
> > > > in my tests:
> > > >
> > > > --load test dataset
> > > > in = LOAD '$pathToIn' using AvroStorage();
> > > >
> > > > --project the fields I need
> > > > inProjected = FOREACH in GENERATE f1, f2;
> > > >
> > > > Right now I can't do this. PigUnit requires explicit field projection
> > on
> > > > LOAD statement.
> > > >
> > >
> >
>
Re: Unoverride load in PigUint
Posted by Ruslan Al-Fakikh <me...@gmail.com>.
It seems to be a heavy PigUnit limitation. Maybe you can open a jira for
this?:)
Thanks,
Ruslan
On Sat, Dec 21, 2013 at 2:01 AM, Serega Sheypak <se...@gmail.com>wrote:
> No :)
> I did extend PigUnit to work with several input relations (i mean 1..N load
> statements), but I didn't find a way to use "native" loader/storage.
> Looks like that the only solution is to create wrapper: data-driven tester
> which feeds script to local pig server and verifies output.
> We did in Megafon. I tried to use "recommended approach" - pig unit for my
> own purposes.
> Fail.
>
>
> 2013/12/21 Ruslan Al-Fakikh <me...@gmail.com>
>
> > Hi Serega!
> >
> > Have you resolved the issue? I am going to encounter the same problem,
> but
> > I don't know a solution.
> >
> > Thanks
> >
> >
> > On Sun, Dec 15, 2013 at 6:07 PM, Serega Sheypak <
> serega.sheypak@gmail.com
> > >wrote:
> >
> > > Hi!
> > > By default PigUnit does override LOAD statements
> > > Is there any possiblity to void this?
> > > I'm using AvroStorage because of evolving schemas and I would like to
> > write
> > > in my tests:
> > >
> > > --load test dataset
> > > in = LOAD '$pathToIn' using AvroStorage();
> > >
> > > --project the fields I need
> > > inProjected = FOREACH in GENERATE f1, f2;
> > >
> > > Right now I can't do this. PigUnit requires explicit field projection
> on
> > > LOAD statement.
> > >
> >
>
Re: Unoverride load in PigUint
Posted by Serega Sheypak <se...@gmail.com>.
No :)
I did extend PigUnit to work with several input relations (i mean 1..N load
statements), but I didn't find a way to use "native" loader/storage.
Looks like that the only solution is to create wrapper: data-driven tester
which feeds script to local pig server and verifies output.
We did in Megafon. I tried to use "recommended approach" - pig unit for my
own purposes.
Fail.
2013/12/21 Ruslan Al-Fakikh <me...@gmail.com>
> Hi Serega!
>
> Have you resolved the issue? I am going to encounter the same problem, but
> I don't know a solution.
>
> Thanks
>
>
> On Sun, Dec 15, 2013 at 6:07 PM, Serega Sheypak <serega.sheypak@gmail.com
> >wrote:
>
> > Hi!
> > By default PigUnit does override LOAD statements
> > Is there any possiblity to void this?
> > I'm using AvroStorage because of evolving schemas and I would like to
> write
> > in my tests:
> >
> > --load test dataset
> > in = LOAD '$pathToIn' using AvroStorage();
> >
> > --project the fields I need
> > inProjected = FOREACH in GENERATE f1, f2;
> >
> > Right now I can't do this. PigUnit requires explicit field projection on
> > LOAD statement.
> >
>
Re: Unoverride load in PigUint
Posted by Ruslan Al-Fakikh <me...@gmail.com>.
Hi Serega!
Have you resolved the issue? I am going to encounter the same problem, but
I don't know a solution.
Thanks
On Sun, Dec 15, 2013 at 6:07 PM, Serega Sheypak <se...@gmail.com>wrote:
> Hi!
> By default PigUnit does override LOAD statements
> Is there any possiblity to void this?
> I'm using AvroStorage because of evolving schemas and I would like to write
> in my tests:
>
> --load test dataset
> in = LOAD '$pathToIn' using AvroStorage();
>
> --project the fields I need
> inProjected = FOREACH in GENERATE f1, f2;
>
> Right now I can't do this. PigUnit requires explicit field projection on
> LOAD statement.
>