You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by yo...@wipro.com on 2012/09/26 07:53:46 UTC
How to use DISTINCT for this case
Hi all,
I am not getting the idea to use distinct for this, or how else we can perform this
I have a file DATAFILE have content like.
Date, time, ssitename, sip, csusername, cip
(2012-01-17,00:04:50,W3SVC1970829965,10.100.195.16,-,180.150.248.195)
(2012-01-17,00:04:50,W3SVC1970829965,10.100.195.16,-,180.150.248.195)
(2012-01-17,00:04:51,W3SVC1970829965,10.100.195.16,-,180.150.248.195)
(2012-01-17,00:04:51,W3SVC1970829965,10.100.195.16,-,180.150.248.195)
(2012-01-17,00:04:51,W3SVC1970829965,10.100.195.16,-,180.150.248.195)
(2012-01-17,00:04:51,W3SVC1970829965,10.100.195.16,-,180.150.248.195)
(2012-01-17,00:04:51,W3SVC1970829965,10.100.195.16,-,180.150.248.195)
(2012-01-17,00:04:56,W3SVC1970829965,10.100.195.16,-,174.6.225.223)
(2012-01-17,00:04:57,W3SVC1970829965,10.100.195.16,-,10.100.195.16)
I did group and count operation with filter condition.
like
A = limit DATAFILE 400;
Datagroup = group A by cip;
Datacount = foreach Datagroup generate group, A.date, COUNT(A) as number;
and got output like this.
(66.249.72.75,{(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17)},7)
(66.249.72.76,{(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17)},7)
(77.88.30.248,{(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17)},5)
(92.97.183.58,{(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17)},4)
(94.242.214.7,{(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17)},5)
............................
.......................
I want it to be as . ( Want to see date appears only at once, not for the number of it repeats as count)
(66.249.72.75,{(2012-01-17)},7)
(66.249.72.76,{(2012-01-17),7)
(77.88.30.248,{(2012-01-17),5)
(92.97.183.58,{(2012-01-17),4)
(94.242.214.7,{(2012-01-17),5)
Please Suggest and Help
Thanks & Regards
Yogesh Kumar
08147912904
Please do not print this email unless it is absolutely necessary.
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.
WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.
www.wipro.com
RE: How to use DISTINCT for this case
Posted by "Manish.Bhoge" <Ma...@target.com>.
Yogesh,
I think to get desired result you need to group by both cid and date , time. If you don't want to do the group by both then break the original dataset into two and keep the date, time separate from the dataset where you are running the group by and later point join them back bases on cid to get the desired result.
Hope this will work.
Thank You,
Manish
-----Original Message-----
From: yogesh.kumar13@wipro.com [mailto:yogesh.kumar13@wipro.com]
Sent: Wednesday, September 26, 2012 11:24 AM
To: user@pig.apache.org
Subject: How to use DISTINCT for this case
Hi all,
I am not getting the idea to use distinct for this, or how else we can perform this
I have a file DATAFILE have content like.
Date, time, ssitename, sip, csusername, cip
(2012-01-17,00:04:50,W3SVC1970829965,10.100.195.16,-,180.150.248.195)
(2012-01-17,00:04:50,W3SVC1970829965,10.100.195.16,-,180.150.248.195)
(2012-01-17,00:04:51,W3SVC1970829965,10.100.195.16,-,180.150.248.195)
(2012-01-17,00:04:51,W3SVC1970829965,10.100.195.16,-,180.150.248.195)
(2012-01-17,00:04:51,W3SVC1970829965,10.100.195.16,-,180.150.248.195)
(2012-01-17,00:04:51,W3SVC1970829965,10.100.195.16,-,180.150.248.195)
(2012-01-17,00:04:51,W3SVC1970829965,10.100.195.16,-,180.150.248.195)
(2012-01-17,00:04:56,W3SVC1970829965,10.100.195.16,-,174.6.225.223)
(2012-01-17,00:04:57,W3SVC1970829965,10.100.195.16,-,10.100.195.16)
I did group and count operation with filter condition.
like
A = limit DATAFILE 400;
Datagroup = group A by cip;
Datacount = foreach Datagroup generate group, A.date, COUNT(A) as number;
and got output like this.
(66.249.72.75,{(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17)},7)
(66.249.72.76,{(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17)},7)
(77.88.30.248,{(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17)},5)
(92.97.183.58,{(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17)},4)
(94.242.214.7,{(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17),(2012-01-17)},5)
............................
.......................
I want it to be as . ( Want to see date appears only at once, not for the number of it repeats as count)
(66.249.72.75,{(2012-01-17)},7)
(66.249.72.76,{(2012-01-17),7)
(77.88.30.248,{(2012-01-17),5)
(92.97.183.58,{(2012-01-17),4)
(94.242.214.7,{(2012-01-17),5)
Please Suggest and Help
Thanks & Regards
Yogesh Kumar
08147912904
Please do not print this email unless it is absolutely necessary.
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.
WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.
www.wipro.com