You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Irooniam <ir...@gmail.com> on 2011/06/18 02:43:53 UTC
regex and query strings
Hello,
I'm having an issue with regex in pig.
Specifically, I'm loading an apache access log and trying to break out the
bits from the query string:
logs = LOAD '$input' using logloader as (remoteHost:CHARARRAY,
hyphen:CHARARRAY, hyphen2:CHARARRAY, time:CHARARRAY, method:CHARARRAY,
uri:CHARARRAY, protocol:CHARARRAY, statusCode:CHARARRAY,
responseSize:CHARARRAY, treferer:CHARARRAY, agent:CHARARRAY);
full_logs = FOREACH logs GENERATE time, uri, FLATTEN(REGEX_EXTRACT(uri,
'id=[0-9]', 2));
The uri looks like:
/khello.html?ref=http%3A%2F%2Fwww.google.com
%2F&k=4165427574dfdb75e0a37a8c13ab757d4273a283&id=1234
However when I run this simple pig script, I get the uri but not the 'id'
parameter.
I then tried using "\d" instead of [0-9] - still won't work.
I tried both [0-9] and \d in php and I get 'id=1' and '1' so I'm not sure
what I'm doing wrong.
Thanks in advance.
Re: regex and query strings
Posted by Irooniam <ir...@gmail.com>.
Awesome, works as advertised.
Thanks for the help Jonathan.
On Fri, Jun 17, 2011 at 6:04 PM, Jonathan Coveney <jc...@gmail.com>wrote:
> regex extract doesn't need to be flattened. In this case, use:
>
> REGEX_EXTRACT(uri,'id=(\\d*)',0); --returns id=1234
> or
> REGEX_EXTRACT(uri,'id=(\\d*)',1); --returns 1234
>
> You were missing the *, which is why it only grabbed the 1.
>
> 2011/6/17 Irooniam <ir...@gmail.com>
>
> > Hello,
> >
> > I'm having an issue with regex in pig.
> >
> > Specifically, I'm loading an apache access log and trying to break out
> the
> > bits from the query string:
> >
> > logs = LOAD '$input' using logloader as (remoteHost:CHARARRAY,
> > hyphen:CHARARRAY, hyphen2:CHARARRAY, time:CHARARRAY, method:CHARARRAY,
> > uri:CHARARRAY, protocol:CHARARRAY, statusCode:CHARARRAY,
> > responseSize:CHARARRAY, treferer:CHARARRAY, agent:CHARARRAY);
> >
> > full_logs = FOREACH logs GENERATE time, uri, FLATTEN(REGEX_EXTRACT(uri,
> > 'id=[0-9]', 2));
> >
> > The uri looks like:
> > /khello.html?ref=http%3A%2F%2Fwww.google.com
> > %2F&k=4165427574dfdb75e0a37a8c13ab757d4273a283&id=1234
> >
> > However when I run this simple pig script, I get the uri but not the 'id'
> > parameter.
> >
> > I then tried using "\d" instead of [0-9] - still won't work.
> >
> > I tried both [0-9] and \d in php and I get 'id=1' and '1' so I'm not sure
> > what I'm doing wrong.
> >
> > Thanks in advance.
> >
>
Re: regex and query strings
Posted by Jonathan Coveney <jc...@gmail.com>.
regex extract doesn't need to be flattened. In this case, use:
REGEX_EXTRACT(uri,'id=(\\d*)',0); --returns id=1234
or
REGEX_EXTRACT(uri,'id=(\\d*)',1); --returns 1234
You were missing the *, which is why it only grabbed the 1.
2011/6/17 Irooniam <ir...@gmail.com>
> Hello,
>
> I'm having an issue with regex in pig.
>
> Specifically, I'm loading an apache access log and trying to break out the
> bits from the query string:
>
> logs = LOAD '$input' using logloader as (remoteHost:CHARARRAY,
> hyphen:CHARARRAY, hyphen2:CHARARRAY, time:CHARARRAY, method:CHARARRAY,
> uri:CHARARRAY, protocol:CHARARRAY, statusCode:CHARARRAY,
> responseSize:CHARARRAY, treferer:CHARARRAY, agent:CHARARRAY);
>
> full_logs = FOREACH logs GENERATE time, uri, FLATTEN(REGEX_EXTRACT(uri,
> 'id=[0-9]', 2));
>
> The uri looks like:
> /khello.html?ref=http%3A%2F%2Fwww.google.com
> %2F&k=4165427574dfdb75e0a37a8c13ab757d4273a283&id=1234
>
> However when I run this simple pig script, I get the uri but not the 'id'
> parameter.
>
> I then tried using "\d" instead of [0-9] - still won't work.
>
> I tried both [0-9] and \d in php and I get 'id=1' and '1' so I'm not sure
> what I'm doing wrong.
>
> Thanks in advance.
>