You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spot.apache.org by Dimitris Papadopoulos <dp...@gmail.com> on 2017/07/08 23:30:05 UTC
Captured netflow v9 fields/tags different from those on public datasets
Hi all,
I 'm posting this here, in case it's more visible than in the general Slack
channel.
We have installed Spot on a testbed (Ubuntu 14.04, CDH 5.11), trying to
simulate a DDoS attack in order to test the platform's detection
capabilities.
We are using a DDoS simulation tool to attack one of our websites, while
capturing netflow traffic (nfcapd) which should normally be ingested and
passed to the hdfs and to hive tables.
Unfortunately, while the flow worker tries to output the nfdump command to
.csv, it fails , probably due to the fact that the netflow fields provided
by our captured traffic are different than those expected.
More specifically, our *nfdump -r -o csv *command outputs files with the
following headers:
ts,te,td,sa,da,sp,dp,pr,flg,fwd,stos,ipkt,ibyt,opkt,obyt,in,out,sas,das,smk,dmk,dtos,dir,nh,nhb,svln,dvln,ismc,odmc,idmc,osmc,mpls1,mpls2,mpls3,mpls4,mpls5,mpls6,mpls7,mpls8,mpls9,mpls10,cl,sl,al,ra,eng,exid,tr
while the public AWS datasets that Spot works with, output just the
following headers:
tr,try,trm,trd,trh,trm,trs,td,sa,da,sp,dp,pr,flg,fwd,stos,ipkt,ibyt,opkt,obyt,in,out,sas,das,dtos,dir,ra
I would like to know the suggested procedure to capture netflow traffic
with the correct format, as it seems that a simple nfcapd command is not
enough.
My colleague is getting the .nfcapd files from a pfsense firewall and he
seems to have matched the correct format (although some issues with the
timestamp of the records have emerged - 1/1/1970 is displayed, probably due
to null values).
I would really appreciate your help, either by replying to this mail, or
via Slack.
Best Regards,
Dimitris
Re: Captured netflow v9 fields/tags different from those on public datasets
Posted by Dimitris Papadopoulos <dp...@gmail.com>.
Thanks a lot Vic!
I was using the regular nfdump version, and I was having problem exporting
to csv. The ONI version appropriately exports our nfcapd FW traffic. :)
Best,
Dimitris
2017-07-09 4:31 GMT+03:00 Gonzalez, Victor <vi...@intel.com>:
> Spot-nfdump is located in the following link
>
> https://github.com/Open-Network-Insight/spot-nfdump
>
> Sent from my iPhone
>
> On Jul 8, 2017, at 6:30 PM, Dimitris Papadopoulos <
> dpapadopoulos91@gmail.com<ma...@gmail.com>> wrote:
>
> Hi all,
>
> I 'm posting this here, in case it's more visible than in the general Slack
> channel.
>
> We have installed Spot on a testbed (Ubuntu 14.04, CDH 5.11), trying to
> simulate a DDoS attack in order to test the platform's detection
> capabilities.
>
> We are using a DDoS simulation tool to attack one of our websites, while
> capturing netflow traffic (nfcapd) which should normally be ingested and
> passed to the hdfs and to hive tables.
>
> Unfortunately, while the flow worker tries to output the nfdump command to
> .csv, it fails , probably due to the fact that the netflow fields provided
> by our captured traffic are different than those expected.
>
> More specifically, our *nfdump -r -o csv *command outputs files with the
> following headers:
> ts,te,td,sa,da,sp,dp,pr,flg,fwd,stos,ipkt,ibyt,opkt,obyt,
> in,out,sas,das,smk,dmk,dtos,dir,nh,nhb,svln,dvln,ismc,
> odmc,idmc,osmc,mpls1,mpls2,mpls3,mpls4,mpls5,mpls6,mpls7,
> mpls8,mpls9,mpls10,cl,sl,al,ra,eng,exid,tr
>
> while the public AWS datasets that Spot works with, output just the
> following headers:
> tr,try,trm,trd,trh,trm,trs,td,sa,da,sp,dp,pr,flg,fwd,stos,
> ipkt,ibyt,opkt,obyt,in,out,sas,das,dtos,dir,ra
>
> I would like to know the suggested procedure to capture netflow traffic
> with the correct format, as it seems that a simple nfcapd command is not
> enough.
> My colleague is getting the .nfcapd files from a pfsense firewall and he
> seems to have matched the correct format (although some issues with the
> timestamp of the records have emerged - 1/1/1970 is displayed, probably due
> to null values).
>
> I would really appreciate your help, either by replying to this mail, or
> via Slack.
>
> Best Regards,
> Dimitris
>
Re: Captured netflow v9 fields/tags different from those on public
datasets
Posted by "Gonzalez, Victor" <vi...@intel.com>.
Spot-nfdump is located in the following link
https://github.com/Open-Network-Insight/spot-nfdump
Sent from my iPhone
On Jul 8, 2017, at 6:30 PM, Dimitris Papadopoulos <dp...@gmail.com>> wrote:
Hi all,
I 'm posting this here, in case it's more visible than in the general Slack
channel.
We have installed Spot on a testbed (Ubuntu 14.04, CDH 5.11), trying to
simulate a DDoS attack in order to test the platform's detection
capabilities.
We are using a DDoS simulation tool to attack one of our websites, while
capturing netflow traffic (nfcapd) which should normally be ingested and
passed to the hdfs and to hive tables.
Unfortunately, while the flow worker tries to output the nfdump command to
.csv, it fails , probably due to the fact that the netflow fields provided
by our captured traffic are different than those expected.
More specifically, our *nfdump -r -o csv *command outputs files with the
following headers:
ts,te,td,sa,da,sp,dp,pr,flg,fwd,stos,ipkt,ibyt,opkt,obyt,in,out,sas,das,smk,dmk,dtos,dir,nh,nhb,svln,dvln,ismc,odmc,idmc,osmc,mpls1,mpls2,mpls3,mpls4,mpls5,mpls6,mpls7,mpls8,mpls9,mpls10,cl,sl,al,ra,eng,exid,tr
while the public AWS datasets that Spot works with, output just the
following headers:
tr,try,trm,trd,trh,trm,trs,td,sa,da,sp,dp,pr,flg,fwd,stos,ipkt,ibyt,opkt,obyt,in,out,sas,das,dtos,dir,ra
I would like to know the suggested procedure to capture netflow traffic
with the correct format, as it seems that a simple nfcapd command is not
enough.
My colleague is getting the .nfcapd files from a pfsense firewall and he
seems to have matched the correct format (although some issues with the
timestamp of the records have emerged - 1/1/1970 is displayed, probably due
to null values).
I would really appreciate your help, either by replying to this mail, or
via Slack.
Best Regards,
Dimitris
Re: Captured netflow v9 fields/tags different from those on public
datasets
Posted by "Gonzalez, Victor" <vi...@intel.com>.
Hi Dimitris,
Are you using spot-nfdump version? Or regular nfcapd/nfdump?
Sent from my iPhone
> On Jul 8, 2017, at 6:30 PM, Dimitris Papadopoulos <dp...@gmail.com> wrote:
>
> Hi all,
>
> I 'm posting this here, in case it's more visible than in the general Slack
> channel.
>
> We have installed Spot on a testbed (Ubuntu 14.04, CDH 5.11), trying to
> simulate a DDoS attack in order to test the platform's detection
> capabilities.
>
> We are using a DDoS simulation tool to attack one of our websites, while
> capturing netflow traffic (nfcapd) which should normally be ingested and
> passed to the hdfs and to hive tables.
>
> Unfortunately, while the flow worker tries to output the nfdump command to
> .csv, it fails , probably due to the fact that the netflow fields provided
> by our captured traffic are different than those expected.
>
> More specifically, our *nfdump -r -o csv *command outputs files with the
> following headers:
> ts,te,td,sa,da,sp,dp,pr,flg,fwd,stos,ipkt,ibyt,opkt,obyt,in,out,sas,das,smk,dmk,dtos,dir,nh,nhb,svln,dvln,ismc,odmc,idmc,osmc,mpls1,mpls2,mpls3,mpls4,mpls5,mpls6,mpls7,mpls8,mpls9,mpls10,cl,sl,al,ra,eng,exid,tr
>
> while the public AWS datasets that Spot works with, output just the
> following headers:
> tr,try,trm,trd,trh,trm,trs,td,sa,da,sp,dp,pr,flg,fwd,stos,ipkt,ibyt,opkt,obyt,in,out,sas,das,dtos,dir,ra
>
> I would like to know the suggested procedure to capture netflow traffic
> with the correct format, as it seems that a simple nfcapd command is not
> enough.
> My colleague is getting the .nfcapd files from a pfsense firewall and he
> seems to have matched the correct format (although some issues with the
> timestamp of the records have emerged - 1/1/1970 is displayed, probably due
> to null values).
>
> I would really appreciate your help, either by replying to this mail, or
> via Slack.
>
> Best Regards,
> Dimitris