You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@pig.apache.org by Tim Chan <tc...@edmunds.com> on 2013/03/21 23:11:07 UTC

filter on a date column?

Since there is not date datatype, how do I filter on a date column?
I've been setting the date column as a chararray.

I would like to do something like:


a = filter b by date_col < '2013-01-01';

Re: filter on a date column?

Posted by Tim Chan <tc...@edmunds.com>.

Hi Cheolsoo,

That worked great.

Thanks,

Tim


On Thu, Mar 21, 2013 at 9:41 PM, Cheolsoo Park <pi...@gmail.com> wrote:

> Hi Tim,
>
> Can't you use ToDate(field, 'format string') instead of casting? For
> example, if I have the following record in my input file:
>
> 2013/01/01
>
> I can do:
>
> a = LOAD 'input.txt' AS (str:chararray);
> b = FOREACH a GENERATE ToDate(str, 'YYYY/MM/DD');
> DUMP b;
>
> This gives me:
>
> (2013-01-01T00:00:00.000-08:00)
>
> Thanks,
> Cheolsoo
>
>
>
> On Thu, Mar 21, 2013 at 9:11 PM, Tim Chan <tc...@edmunds.com> wrote:
>
> > Hi Cheolsoo,
> >
> > What if my date time column is not formatted properly?
> > I noticed when I cast my column to datetime, it comes out with null
> values.
> >
> > Thanks,
> >
> > Tim
> >
> >
> > On Thu, Mar 21, 2013 at 4:46 PM, Cheolsoo Park <pi...@gmail.com>
> > wrote:
> >
> > > Hi Tim,
> > >
> > > Pig 0.11 has datetime type. You can do something like this:
> > >
> > > a = LOAD 'input.txt' AS (date:datetime);
> > > b = FILTER a BY date < ToDate('2013-01-01');
> > >
> > > Also see built-in functions for datetime type:
> > > http://pig.apache.org/docs/r0.11.0/func.html#datetime-functions
> > >
> > > Thanks,
> > > Cheolsoo
> > >
> > >
> > > On Thu, Mar 21, 2013 at 3:11 PM, Tim Chan <tc...@edmunds.com> wrote:
> > >
> > > > Since there is not date datatype, how do I filter on a date column?
> > > > I've been setting the date column as a chararray.
> > > >
> > > > I would like to do something like:
> > > >
> > > >
> > > > a = filter b by date_col < '2013-01-01';
> > > >
> > >
> >
>

Re: filter on a date column?

Posted by Cheolsoo Park <pi...@gmail.com>.

Hi Tim,

Can't you use ToDate(field, 'format string') instead of casting? For
example, if I have the following record in my input file:

2013/01/01

I can do:

a = LOAD 'input.txt' AS (str:chararray);
b = FOREACH a GENERATE ToDate(str, 'YYYY/MM/DD');
DUMP b;

This gives me:

(2013-01-01T00:00:00.000-08:00)

Thanks,
Cheolsoo



On Thu, Mar 21, 2013 at 9:11 PM, Tim Chan <tc...@edmunds.com> wrote:

> Hi Cheolsoo,
>
> What if my date time column is not formatted properly?
> I noticed when I cast my column to datetime, it comes out with null values.
>
> Thanks,
>
> Tim
>
>
> On Thu, Mar 21, 2013 at 4:46 PM, Cheolsoo Park <pi...@gmail.com>
> wrote:
>
> > Hi Tim,
> >
> > Pig 0.11 has datetime type. You can do something like this:
> >
> > a = LOAD 'input.txt' AS (date:datetime);
> > b = FILTER a BY date < ToDate('2013-01-01');
> >
> > Also see built-in functions for datetime type:
> > http://pig.apache.org/docs/r0.11.0/func.html#datetime-functions
> >
> > Thanks,
> > Cheolsoo
> >
> >
> > On Thu, Mar 21, 2013 at 3:11 PM, Tim Chan <tc...@edmunds.com> wrote:
> >
> > > Since there is not date datatype, how do I filter on a date column?
> > > I've been setting the date column as a chararray.
> > >
> > > I would like to do something like:
> > >
> > >
> > > a = filter b by date_col < '2013-01-01';
> > >
> >
>

Re: filter on a date column?

Posted by Tim Chan <tc...@edmunds.com>.

Hi Cheolsoo,

What if my date time column is not formatted properly?
I noticed when I cast my column to datetime, it comes out with null values.

Thanks,

Tim


On Thu, Mar 21, 2013 at 4:46 PM, Cheolsoo Park <pi...@gmail.com> wrote:

> Hi Tim,
>
> Pig 0.11 has datetime type. You can do something like this:
>
> a = LOAD 'input.txt' AS (date:datetime);
> b = FILTER a BY date < ToDate('2013-01-01');
>
> Also see built-in functions for datetime type:
> http://pig.apache.org/docs/r0.11.0/func.html#datetime-functions
>
> Thanks,
> Cheolsoo
>
>
> On Thu, Mar 21, 2013 at 3:11 PM, Tim Chan <tc...@edmunds.com> wrote:
>
> > Since there is not date datatype, how do I filter on a date column?
> > I've been setting the date column as a chararray.
> >
> > I would like to do something like:
> >
> >
> > a = filter b by date_col < '2013-01-01';
> >
>

Re: filter on a date column?

Posted by Cheolsoo Park <pi...@gmail.com>.

Hi Tim,

Pig 0.11 has datetime type. You can do something like this:

a = LOAD 'input.txt' AS (date:datetime);
b = FILTER a BY date < ToDate('2013-01-01');

Also see built-in functions for datetime type:
http://pig.apache.org/docs/r0.11.0/func.html#datetime-functions

Thanks,
Cheolsoo


On Thu, Mar 21, 2013 at 3:11 PM, Tim Chan <tc...@edmunds.com> wrote:

> Since there is not date datatype, how do I filter on a date column?
> I've been setting the date column as a chararray.
>
> I would like to do something like:
>
>
> a = filter b by date_col < '2013-01-01';
>