Pig Script for any dataset (Doubts and suggestions)

[rahul raghavendhra <ra...@gmail.com>]

I have a plan to write a simple classification algorithm on any csv using
Pig and embed py..
 Can i write  generic pig scripts that apply for any dataset..?
 Is there any guidelines or examples for generic pig scipts which works for
any dataset(number of colums of the dataset also vary )..
can use all the python statements within that py file.. ?
please help..

Re: Pig Script for any dataset (Doubts and suggestions)

[rahul raghavendhra <ra...@gmail.com>]

it just involves grouping the all other attribute with class attribute and
calculate the count and calculate probability..

all i want is to pass the arguments to the pig py script..
can pass arguments to py file ?

i tried like
>pig sample.py dataset.csv number of attributes(column count)

but i cant access it using sys.argv
its end up with error..

please help
./rahul


On Thu, Jan 12, 2012 at 2:42 PM, Prashant Kommireddi <pr...@gmail.com>wrote:

> It depends on the logic you would like to implement in your script. How
> does the algorithm treat additional columns?
> If the behavior is consistent between the columns, for eg if the script was
> an additive function like ColA + ColB, and then adding a column was as
> simple as ColA + ColB + ColC that could be automated into your Pig script.
>
> Thanks,
> Prashant
>
> On Thu, Jan 12, 2012 at 1:01 AM, rahul raghavendhra <
> rahulraghavendhra1@gmail.com> wrote:
>
> > I have a plan to write a simple classification algorithm on any csv using
> > Pig and embed py..
> >  Can i write  generic pig scripts that apply for any dataset..?
> >  Is there any guidelines or examples for generic pig scipts which works
> for
> > any dataset(number of colums of the dataset also vary )..
> > can use all the python statements within that py file.. ?
> > please help..
> >
>

Re: Pig Script for any dataset (Doubts and suggestions)

[Prashant Kommireddi <pr...@gmail.com>]

It depends on the logic you would like to implement in your script. How
does the algorithm treat additional columns?
If the behavior is consistent between the columns, for eg if the script was
an additive function like ColA + ColB, and then adding a column was as
simple as ColA + ColB + ColC that could be automated into your Pig script.

Thanks,
Prashant

On Thu, Jan 12, 2012 at 1:01 AM, rahul raghavendhra <
rahulraghavendhra1@gmail.com> wrote:

> I have a plan to write a simple classification algorithm on any csv using
> Pig and embed py..
>  Can i write  generic pig scripts that apply for any dataset..?
>  Is there any guidelines or examples for generic pig scipts which works for
> any dataset(number of colums of the dataset also vary )..
> can use all the python statements within that py file.. ?
> please help..
>