You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Charles Givre <cg...@gmail.com> on 2018/01/01 20:31:17 UTC
PCAP Issues
Hello all,
I was playing with the PCAP functionality in Drill and I wanted to add the TCP flags to the data that Drill is returning. I was also interested in adding the TCP Sequence and Ack numbers as well. I noticed that the code as written currently has a function in Packet.java which returns the TCP Sequence number, however this was never added to the schema, so I added that and rebuilt Drill, however, it doesn’t seem to be returning the correct result. The file I was querying is attached to this email, and should in all cases return a sequence number of zero.
Questions:
1. Could someone please take a look at the code for the tcp_sequence and see if I did something wrong, or if the offset is not being calculated correctly
2. I’m trying to figure out the offsets for the various TCP flags. I would think that the offset should be PacketConstants.ETHER_HEADER_LENGTH + getIPHeaderLength() +13 to get the word that has the flags and then from there, access the individual bits. However, this doesn’t seem to work. What am I missing?
Thanks and Happy New Year!
- C
Re: PCAP Issues
Posted by Ted Dunning <td...@mapr.com>.
And what I said only applies for sure to IPV4. I don't know if the TCP structure is the same for IPV6.
________________________________
From: Ted Dunning
Sent: Monday, January 1, 2018 10:42:15 PM
To: Charles Givre; dev@drill.apache.org
Subject: Re: PCAP Issues
PacketConstants.ETHER_HEADER_LENGTH + getIPHeaderLength() +13 to get the word that has the flags
Looks to me like
getByte(raw, ipOffset + getIPHeaderLength() + 13)
is what you need. And this gets you the byte, not the word.
________________________________
From: Charles Givre <cg...@gmail.com>
Sent: Monday, January 1, 2018 12:31:17 PM
To: dev@drill.apache.org
Cc: Ted Dunning
Subject: PCAP Issues
Hello all,
I was playing with the PCAP functionality in Drill and I wanted to add the TCP flags to the data that Drill is returning. I was also interested in adding the TCP Sequence and Ack numbers as well. I noticed that the code as written currently has a function in Packet.java which returns the TCP Sequence number, however this was never added to the schema, so I added that and rebuilt Drill, however, it doesn’t seem to be returning the correct result. The file I was querying is attached to this email, and should in all cases return a sequence number of zero.
Questions:
1. Could someone please take a look at the code for the tcp_sequence and see if I did something wrong, or if the offset is not being calculated correctly
2. I’m trying to figure out the offsets for the various TCP flags. I would think that the offset should be PacketConstants.ETHER_HEADER_LENGTH + getIPHeaderLength() +13 to get the word that has the flags and then from there, access the individual bits. However, this doesn’t seem to work. What am I missing?
Thanks and Happy New Year!
- C
Re: PCAP Issues
Posted by Ted Dunning <td...@mapr.com>.
PacketConstants.ETHER_HEADER_LENGTH + getIPHeaderLength() +13 to get the word that has the flags
Looks to me like
getByte(raw, ipOffset + getIPHeaderLength() + 13)
is what you need. And this gets you the byte, not the word.
________________________________
From: Charles Givre <cg...@gmail.com>
Sent: Monday, January 1, 2018 12:31:17 PM
To: dev@drill.apache.org
Cc: Ted Dunning
Subject: PCAP Issues
Hello all,
I was playing with the PCAP functionality in Drill and I wanted to add the TCP flags to the data that Drill is returning. I was also interested in adding the TCP Sequence and Ack numbers as well. I noticed that the code as written currently has a function in Packet.java which returns the TCP Sequence number, however this was never added to the schema, so I added that and rebuilt Drill, however, it doesn’t seem to be returning the correct result. The file I was querying is attached to this email, and should in all cases return a sequence number of zero.
Questions:
1. Could someone please take a look at the code for the tcp_sequence and see if I did something wrong, or if the offset is not being calculated correctly
2. I’m trying to figure out the offsets for the various TCP flags. I would think that the offset should be PacketConstants.ETHER_HEADER_LENGTH + getIPHeaderLength() +13 to get the word that has the flags and then from there, access the individual bits. However, this doesn’t seem to work. What am I missing?
Thanks and Happy New Year!
- C
Re: PCAP Issues
Posted by Ted Dunning <td...@mapr.com>.
2. I’m trying to figure out the offsets for the various TCP flags. I
>> would think that the offset should be PacketConstants.ETHER_HEADER_LENGTH
>> + getIPHeaderLength() +13
Looks to me like the offset here is correct, and adding the IP header length looks right. I also remember that this was finicky in the implementation. I will have to take a look.
________________________________
From: Charles Givre <cg...@gmail.com>
Sent: Monday, January 1, 2018 12:31:17 PM
To: dev@drill.apache.org
Cc: Ted Dunning
Subject: PCAP Issues
Hello all,
I was playing with the PCAP functionality in Drill and I wanted to add the TCP flags to the data that Drill is returning. I was also interested in adding the TCP Sequence and Ack numbers as well. I noticed that the code as written currently has a function in Packet.java which returns the TCP Sequence number, however this was never added to the schema, so I added that and rebuilt Drill, however, it doesn’t seem to be returning the correct result. The file I was querying is attached to this email, and should in all cases return a sequence number of zero.
Questions:
1. Could someone please take a look at the code for the tcp_sequence and see if I did something wrong, or if the offset is not being calculated correctly
2. I’m trying to figure out the offsets for the various TCP flags. I would think that the offset should be PacketConstants.ETHER_HEADER_LENGTH + getIPHeaderLength() +13 to get the word that has the flags and then from there, access the individual bits. However, this doesn’t seem to work. What am I missing?
Thanks and Happy New Year!
- C
Re: PCAP Issues
Posted by Ted Dunning <te...@gmail.com>.
The pcap format is there almost entirely just to give us a record structure.
The stream could give us that just as easily.
So that is one thing that could be done.
Another thought is that data could be buffered a (very) short time and
groups of packets could be sent as a single message. That would decrease
streaming overhead. I would vote against that change at first, but it would
make the pcap format relevant again.
On Tue, Jan 9, 2018 at 6:45 AM, John Omernik <jo...@omernik.com> wrote:
> I've been following this and and also now playing with them more. A couple
> of tidbits that I think are worth re-mentioning:
>
> 1. pcapng the ability to read this too will be handy.
>
> 2. The flags are really cool, and the work here is great.
>
> The other thing I'd like to consider for this is with recent work on the
> Kafka plugin, and I am guessing a similar plugin with MapR Streams, what if
> we were able to have a tool like daemonlogger, or tcpdump read from a
> network interface and write directly to a Kafka/MapR Stream topic?
>
> My thought is this, you write to the topic, you partition based on a hash
> of known data components (like the 5-tuple of protocol, sport, dport,
> saddr, and daddr). From here, you could have things subscribe to the topic
> and process, you could etl, you could query. There would be some concern
> at high scale, but even that has methods to address. One of the challenges
> we have infosec is getting processing close enough to capture.... OR
> capture to file, move them around, have creates process to read and process
> files.
>
> It would be cool, to combine the pcap plugin directly with the kafka/stream
> plugin in drill and be able to read live pcaps!
>
> One big challenge is the pcap format has a header, but it seems to be a
> simple header that we "may" be able to forge upon dropping the needle on a
> stream of packet records. I am just brainstorming aloud right now, (I
> should consider waiting until at least 3 cups of coffee before I make this
> inner dialog public)
>
> Curious on thoughts here.
>
>
>
>
> On Fri, Jan 5, 2018 at 10:01 AM, Charles Givre <cg...@gmail.com> wrote:
>
> > I assumed (incorrectly as it turned out) that I was not reading the
> > offsets correctly or something.
> >
> >
> > > On Jan 5, 2018, at 10:59, Ted Dunning <te...@gmail.com> wrote:
> > >
> > > Yeah... I got the same result.
> > >
> > > I will push my latest shortly (without the boolean stuff)
> > >
> > > On Fri, Jan 5, 2018 at 7:52 AM, Charles Givre <cg...@gmail.com>
> wrote:
> > >
> > >> Ted,
> > >> I’m wondering if we’ve uncovered an unsupported, undocumented feature
> in
> > >> Drill with respect to the BitWriter. I made the following changes:
> > >>
> > >>
> > >> To PcapRecordReader.java, I added:
> > >>
> > >>
> > >> case "ack_flag":
> > >> if (packet.isTcpPacket()) {
> > >> int flag = (packet.getAckFlag()) ? 1 : 0;
> > >> setBooleanColumnValue(flag, pci, count);
> > >> }
> > >> break;
> > >> ...
> > >>
> > >> private void setBooleanColumnValue(final int data, final
> > >> ProjectedColumnInfo pci, final int count) {
> > >> ((NullableBitVector.Mutator) pci.vv.getMutator())
> > >> .setSafe(count, data);
> > >> }
> > >>
> > >> Then I added this to PcapFormatUtils
> > >>
> > >> public static int getBit(final byte[] buf, final int posByte, final
> int
> > >> posBit ) {
> > >> byte valByte = buf[posByte];
> > >> return valByte >> (8 - (posBit + 1)) & 0x0001;
> > >> }
> > >>
> > >> I added a column in the Schema.java and a function in Packet.java to
> get
> > >> the bit. I was getting NPEs and I’m wondering now if the cause wasn’t
> > my
> > >> code but rather a problem with the bitwriter. I’m going to play with
> > this
> > >> and see if I can get Drill to write a True/False value at all, and
> > report
> > >> back. If the BitWriter is throwing NPE, I’ll create a JIRA for it.
> > >>
> > >> Thanks for your work on the PCAP format reader.
> > >> —C
> > >>
> > >>
> > >>
> > >>> On Jan 3, 2018, at 17:33, Ted Dunning <te...@gmail.com> wrote:
> > >>>
> > >>> Don't think that will work.
> > >>>
> > >>> I tried what seemed obvious and got a NPE. Joys of Drill.
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Jan 3, 2018 at 1:31 PM, Charles Givre <cg...@gmail.com>
> > wrote:
> > >>>
> > >>>> This isn’t the most elegant example, but should do the trick.
> > >>>>
> > >>>> https://github.com/cgivre/drill-network-functions/blob/
> > >>>> master/src/main/java/org/apache/drill/contrib/function/
> > IsPrivateIP.java
> > >> <
> > >>>> https://github.com/cgivre/drill-network-functions/blob/
> > >>>> master/src/main/java/org/apache/drill/contrib/function/
> > >> IsPrivateIP.java>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>> On Jan 3, 2018, at 16:09, Ted Dunning <te...@gmail.com>
> wrote:
> > >>>>>
> > >>>>> On Wed, Jan 3, 2018 at 12:17 PM, Charles Givre <cg...@gmail.com>
> > >> wrote:
> > >>>>>
> > >>>>>> HI Ted,
> > >>>>>> This is really looking amazing. Thank you so much for working on
> > >>>> this. I
> > >>>>>> wanted to ask whether you’ve tried the BitWriter to write a
> Boolean
> > >>>> value?
> > >>>>>> I’ve done that in UDFs.
> > >>>>>
> > >>>>>
> > >>>>> Point me to an example. I think booleans would be vastly better.
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>
Re: PCAP Issues
Posted by John Omernik <jo...@omernik.com>.
I've been following this and and also now playing with them more. A couple
of tidbits that I think are worth re-mentioning:
1. pcapng the ability to read this too will be handy.
2. The flags are really cool, and the work here is great.
The other thing I'd like to consider for this is with recent work on the
Kafka plugin, and I am guessing a similar plugin with MapR Streams, what if
we were able to have a tool like daemonlogger, or tcpdump read from a
network interface and write directly to a Kafka/MapR Stream topic?
My thought is this, you write to the topic, you partition based on a hash
of known data components (like the 5-tuple of protocol, sport, dport,
saddr, and daddr). From here, you could have things subscribe to the topic
and process, you could etl, you could query. There would be some concern
at high scale, but even that has methods to address. One of the challenges
we have infosec is getting processing close enough to capture.... OR
capture to file, move them around, have creates process to read and process
files.
It would be cool, to combine the pcap plugin directly with the kafka/stream
plugin in drill and be able to read live pcaps!
One big challenge is the pcap format has a header, but it seems to be a
simple header that we "may" be able to forge upon dropping the needle on a
stream of packet records. I am just brainstorming aloud right now, (I
should consider waiting until at least 3 cups of coffee before I make this
inner dialog public)
Curious on thoughts here.
On Fri, Jan 5, 2018 at 10:01 AM, Charles Givre <cg...@gmail.com> wrote:
> I assumed (incorrectly as it turned out) that I was not reading the
> offsets correctly or something.
>
>
> > On Jan 5, 2018, at 10:59, Ted Dunning <te...@gmail.com> wrote:
> >
> > Yeah... I got the same result.
> >
> > I will push my latest shortly (without the boolean stuff)
> >
> > On Fri, Jan 5, 2018 at 7:52 AM, Charles Givre <cg...@gmail.com> wrote:
> >
> >> Ted,
> >> I’m wondering if we’ve uncovered an unsupported, undocumented feature in
> >> Drill with respect to the BitWriter. I made the following changes:
> >>
> >>
> >> To PcapRecordReader.java, I added:
> >>
> >>
> >> case "ack_flag":
> >> if (packet.isTcpPacket()) {
> >> int flag = (packet.getAckFlag()) ? 1 : 0;
> >> setBooleanColumnValue(flag, pci, count);
> >> }
> >> break;
> >> ...
> >>
> >> private void setBooleanColumnValue(final int data, final
> >> ProjectedColumnInfo pci, final int count) {
> >> ((NullableBitVector.Mutator) pci.vv.getMutator())
> >> .setSafe(count, data);
> >> }
> >>
> >> Then I added this to PcapFormatUtils
> >>
> >> public static int getBit(final byte[] buf, final int posByte, final int
> >> posBit ) {
> >> byte valByte = buf[posByte];
> >> return valByte >> (8 - (posBit + 1)) & 0x0001;
> >> }
> >>
> >> I added a column in the Schema.java and a function in Packet.java to get
> >> the bit. I was getting NPEs and I’m wondering now if the cause wasn’t
> my
> >> code but rather a problem with the bitwriter. I’m going to play with
> this
> >> and see if I can get Drill to write a True/False value at all, and
> report
> >> back. If the BitWriter is throwing NPE, I’ll create a JIRA for it.
> >>
> >> Thanks for your work on the PCAP format reader.
> >> —C
> >>
> >>
> >>
> >>> On Jan 3, 2018, at 17:33, Ted Dunning <te...@gmail.com> wrote:
> >>>
> >>> Don't think that will work.
> >>>
> >>> I tried what seemed obvious and got a NPE. Joys of Drill.
> >>>
> >>>
> >>>
> >>> On Wed, Jan 3, 2018 at 1:31 PM, Charles Givre <cg...@gmail.com>
> wrote:
> >>>
> >>>> This isn’t the most elegant example, but should do the trick.
> >>>>
> >>>> https://github.com/cgivre/drill-network-functions/blob/
> >>>> master/src/main/java/org/apache/drill/contrib/function/
> IsPrivateIP.java
> >> <
> >>>> https://github.com/cgivre/drill-network-functions/blob/
> >>>> master/src/main/java/org/apache/drill/contrib/function/
> >> IsPrivateIP.java>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> On Jan 3, 2018, at 16:09, Ted Dunning <te...@gmail.com> wrote:
> >>>>>
> >>>>> On Wed, Jan 3, 2018 at 12:17 PM, Charles Givre <cg...@gmail.com>
> >> wrote:
> >>>>>
> >>>>>> HI Ted,
> >>>>>> This is really looking amazing. Thank you so much for working on
> >>>> this. I
> >>>>>> wanted to ask whether you’ve tried the BitWriter to write a Boolean
> >>>> value?
> >>>>>> I’ve done that in UDFs.
> >>>>>
> >>>>>
> >>>>> Point me to an example. I think booleans would be vastly better.
> >>>>
> >>>>
> >>
> >>
>
>
Re: PCAP Issues
Posted by Charles Givre <cg...@gmail.com>.
I assumed (incorrectly as it turned out) that I was not reading the offsets correctly or something.
> On Jan 5, 2018, at 10:59, Ted Dunning <te...@gmail.com> wrote:
>
> Yeah... I got the same result.
>
> I will push my latest shortly (without the boolean stuff)
>
> On Fri, Jan 5, 2018 at 7:52 AM, Charles Givre <cg...@gmail.com> wrote:
>
>> Ted,
>> I’m wondering if we’ve uncovered an unsupported, undocumented feature in
>> Drill with respect to the BitWriter. I made the following changes:
>>
>>
>> To PcapRecordReader.java, I added:
>>
>>
>> case "ack_flag":
>> if (packet.isTcpPacket()) {
>> int flag = (packet.getAckFlag()) ? 1 : 0;
>> setBooleanColumnValue(flag, pci, count);
>> }
>> break;
>> ...
>>
>> private void setBooleanColumnValue(final int data, final
>> ProjectedColumnInfo pci, final int count) {
>> ((NullableBitVector.Mutator) pci.vv.getMutator())
>> .setSafe(count, data);
>> }
>>
>> Then I added this to PcapFormatUtils
>>
>> public static int getBit(final byte[] buf, final int posByte, final int
>> posBit ) {
>> byte valByte = buf[posByte];
>> return valByte >> (8 - (posBit + 1)) & 0x0001;
>> }
>>
>> I added a column in the Schema.java and a function in Packet.java to get
>> the bit. I was getting NPEs and I’m wondering now if the cause wasn’t my
>> code but rather a problem with the bitwriter. I’m going to play with this
>> and see if I can get Drill to write a True/False value at all, and report
>> back. If the BitWriter is throwing NPE, I’ll create a JIRA for it.
>>
>> Thanks for your work on the PCAP format reader.
>> —C
>>
>>
>>
>>> On Jan 3, 2018, at 17:33, Ted Dunning <te...@gmail.com> wrote:
>>>
>>> Don't think that will work.
>>>
>>> I tried what seemed obvious and got a NPE. Joys of Drill.
>>>
>>>
>>>
>>> On Wed, Jan 3, 2018 at 1:31 PM, Charles Givre <cg...@gmail.com> wrote:
>>>
>>>> This isn’t the most elegant example, but should do the trick.
>>>>
>>>> https://github.com/cgivre/drill-network-functions/blob/
>>>> master/src/main/java/org/apache/drill/contrib/function/IsPrivateIP.java
>> <
>>>> https://github.com/cgivre/drill-network-functions/blob/
>>>> master/src/main/java/org/apache/drill/contrib/function/
>> IsPrivateIP.java>
>>>>
>>>>
>>>>
>>>>
>>>>> On Jan 3, 2018, at 16:09, Ted Dunning <te...@gmail.com> wrote:
>>>>>
>>>>> On Wed, Jan 3, 2018 at 12:17 PM, Charles Givre <cg...@gmail.com>
>> wrote:
>>>>>
>>>>>> HI Ted,
>>>>>> This is really looking amazing. Thank you so much for working on
>>>> this. I
>>>>>> wanted to ask whether you’ve tried the BitWriter to write a Boolean
>>>> value?
>>>>>> I’ve done that in UDFs.
>>>>>
>>>>>
>>>>> Point me to an example. I think booleans would be vastly better.
>>>>
>>>>
>>
>>
Re: PCAP Issues
Posted by Ted Dunning <te...@gmail.com>.
Yeah... I got the same result.
I will push my latest shortly (without the boolean stuff)
On Fri, Jan 5, 2018 at 7:52 AM, Charles Givre <cg...@gmail.com> wrote:
> Ted,
> I’m wondering if we’ve uncovered an unsupported, undocumented feature in
> Drill with respect to the BitWriter. I made the following changes:
>
>
> To PcapRecordReader.java, I added:
>
>
> case "ack_flag":
> if (packet.isTcpPacket()) {
> int flag = (packet.getAckFlag()) ? 1 : 0;
> setBooleanColumnValue(flag, pci, count);
> }
> break;
> ...
>
> private void setBooleanColumnValue(final int data, final
> ProjectedColumnInfo pci, final int count) {
> ((NullableBitVector.Mutator) pci.vv.getMutator())
> .setSafe(count, data);
> }
>
> Then I added this to PcapFormatUtils
>
> public static int getBit(final byte[] buf, final int posByte, final int
> posBit ) {
> byte valByte = buf[posByte];
> return valByte >> (8 - (posBit + 1)) & 0x0001;
> }
>
> I added a column in the Schema.java and a function in Packet.java to get
> the bit. I was getting NPEs and I’m wondering now if the cause wasn’t my
> code but rather a problem with the bitwriter. I’m going to play with this
> and see if I can get Drill to write a True/False value at all, and report
> back. If the BitWriter is throwing NPE, I’ll create a JIRA for it.
>
> Thanks for your work on the PCAP format reader.
> —C
>
>
>
> > On Jan 3, 2018, at 17:33, Ted Dunning <te...@gmail.com> wrote:
> >
> > Don't think that will work.
> >
> > I tried what seemed obvious and got a NPE. Joys of Drill.
> >
> >
> >
> > On Wed, Jan 3, 2018 at 1:31 PM, Charles Givre <cg...@gmail.com> wrote:
> >
> >> This isn’t the most elegant example, but should do the trick.
> >>
> >> https://github.com/cgivre/drill-network-functions/blob/
> >> master/src/main/java/org/apache/drill/contrib/function/IsPrivateIP.java
> <
> >> https://github.com/cgivre/drill-network-functions/blob/
> >> master/src/main/java/org/apache/drill/contrib/function/
> IsPrivateIP.java>
> >>
> >>
> >>
> >>
> >>> On Jan 3, 2018, at 16:09, Ted Dunning <te...@gmail.com> wrote:
> >>>
> >>> On Wed, Jan 3, 2018 at 12:17 PM, Charles Givre <cg...@gmail.com>
> wrote:
> >>>
> >>>> HI Ted,
> >>>> This is really looking amazing. Thank you so much for working on
> >> this. I
> >>>> wanted to ask whether you’ve tried the BitWriter to write a Boolean
> >> value?
> >>>> I’ve done that in UDFs.
> >>>
> >>>
> >>> Point me to an example. I think booleans would be vastly better.
> >>
> >>
>
>
Re: PCAP Issues
Posted by Charles Givre <cg...@gmail.com>.
Ted,
I’m wondering if we’ve uncovered an unsupported, undocumented feature in Drill with respect to the BitWriter. I made the following changes:
To PcapRecordReader.java, I added:
case "ack_flag":
if (packet.isTcpPacket()) {
int flag = (packet.getAckFlag()) ? 1 : 0;
setBooleanColumnValue(flag, pci, count);
}
break;
...
private void setBooleanColumnValue(final int data, final ProjectedColumnInfo pci, final int count) {
((NullableBitVector.Mutator) pci.vv.getMutator())
.setSafe(count, data);
}
Then I added this to PcapFormatUtils
public static int getBit(final byte[] buf, final int posByte, final int posBit ) {
byte valByte = buf[posByte];
return valByte >> (8 - (posBit + 1)) & 0x0001;
}
I added a column in the Schema.java and a function in Packet.java to get the bit. I was getting NPEs and I’m wondering now if the cause wasn’t my code but rather a problem with the bitwriter. I’m going to play with this and see if I can get Drill to write a True/False value at all, and report back. If the BitWriter is throwing NPE, I’ll create a JIRA for it.
Thanks for your work on the PCAP format reader.
—C
> On Jan 3, 2018, at 17:33, Ted Dunning <te...@gmail.com> wrote:
>
> Don't think that will work.
>
> I tried what seemed obvious and got a NPE. Joys of Drill.
>
>
>
> On Wed, Jan 3, 2018 at 1:31 PM, Charles Givre <cg...@gmail.com> wrote:
>
>> This isn’t the most elegant example, but should do the trick.
>>
>> https://github.com/cgivre/drill-network-functions/blob/
>> master/src/main/java/org/apache/drill/contrib/function/IsPrivateIP.java <
>> https://github.com/cgivre/drill-network-functions/blob/
>> master/src/main/java/org/apache/drill/contrib/function/IsPrivateIP.java>
>>
>>
>>
>>
>>> On Jan 3, 2018, at 16:09, Ted Dunning <te...@gmail.com> wrote:
>>>
>>> On Wed, Jan 3, 2018 at 12:17 PM, Charles Givre <cg...@gmail.com> wrote:
>>>
>>>> HI Ted,
>>>> This is really looking amazing. Thank you so much for working on
>> this. I
>>>> wanted to ask whether you’ve tried the BitWriter to write a Boolean
>> value?
>>>> I’ve done that in UDFs.
>>>
>>>
>>> Point me to an example. I think booleans would be vastly better.
>>
>>
Re: PCAP Issues
Posted by Ted Dunning <te...@gmail.com>.
Don't think that will work.
I tried what seemed obvious and got a NPE. Joys of Drill.
On Wed, Jan 3, 2018 at 1:31 PM, Charles Givre <cg...@gmail.com> wrote:
> This isn’t the most elegant example, but should do the trick.
>
> https://github.com/cgivre/drill-network-functions/blob/
> master/src/main/java/org/apache/drill/contrib/function/IsPrivateIP.java <
> https://github.com/cgivre/drill-network-functions/blob/
> master/src/main/java/org/apache/drill/contrib/function/IsPrivateIP.java>
>
>
>
>
> > On Jan 3, 2018, at 16:09, Ted Dunning <te...@gmail.com> wrote:
> >
> > On Wed, Jan 3, 2018 at 12:17 PM, Charles Givre <cg...@gmail.com> wrote:
> >
> >> HI Ted,
> >> This is really looking amazing. Thank you so much for working on
> this. I
> >> wanted to ask whether you’ve tried the BitWriter to write a Boolean
> value?
> >> I’ve done that in UDFs.
> >
> >
> > Point me to an example. I think booleans would be vastly better.
>
>
Re: PCAP Issues
Posted by Charles Givre <cg...@gmail.com>.
This isn’t the most elegant example, but should do the trick.
https://github.com/cgivre/drill-network-functions/blob/master/src/main/java/org/apache/drill/contrib/function/IsPrivateIP.java <https://github.com/cgivre/drill-network-functions/blob/master/src/main/java/org/apache/drill/contrib/function/IsPrivateIP.java>
> On Jan 3, 2018, at 16:09, Ted Dunning <te...@gmail.com> wrote:
>
> On Wed, Jan 3, 2018 at 12:17 PM, Charles Givre <cg...@gmail.com> wrote:
>
>> HI Ted,
>> This is really looking amazing. Thank you so much for working on this. I
>> wanted to ask whether you’ve tried the BitWriter to write a Boolean value?
>> I’ve done that in UDFs.
>
>
> Point me to an example. I think booleans would be vastly better.
Re: PCAP Issues
Posted by Ted Dunning <te...@gmail.com>.
On Wed, Jan 3, 2018 at 12:17 PM, Charles Givre <cg...@gmail.com> wrote:
> HI Ted,
> This is really looking amazing. Thank you so much for working on this. I
> wanted to ask whether you’ve tried the BitWriter to write a Boolean value?
> I’ve done that in UDFs.
Point me to an example. I think booleans would be vastly better.
Re: PCAP Issues
Posted by Charles Givre <cg...@gmail.com>.
HI Ted,
This is really looking amazing. Thank you so much for working on this. I wanted to ask whether you’ve tried the BitWriter to write a Boolean value? I’ve done that in UDFs.
— C
> On Jan 3, 2018, at 14:22, Ted Dunning <te...@gmail.com> wrote:
>
> Added exploded flag fields per Charles suggestion. I couldn't figure out
> how to get a boolean value, however. Will push to the pull request shortly.
>
> 0: jdbc:drill:zk=local> select src_port, dst_port, tcp_session, tcp_ack,
> tcp_flags, tcp_parsed_flags from
> dfs.root.`/Users/tdunning/tmp/synscan.pcap` where tcp_flags_ack>0 limit 10;
>
> *+-----------+-----------+-----------------------+-------------+------------+-------------------+*
>
> *| **src_port ** | **dst_port ** | ** tcp_session ** | ** tcp_ack
> ** | **tcp_flags ** | **tcp_parsed_flags ** |*
>
> *+-----------+-----------+-----------------------+-------------+------------+-------------------+*
>
> *| *53 * | *36050 * | *6346604732028469374 * | *-581795047 * | *18
> * | *ACK|SYN * |*
>
> *| *113 * | *36050 * | *2591649313338334401 * | *-581795047 * | *20
> * | *ACK|RST * |*
>
> *| *80 * | *36050 * | *-9031405983396365775 * | *-581795047 * | *18
> * | *ACK|SYN * |*
>
> *| *22 * | *36050 * | *7738739733723725373 * | *-581795047 * | *18
> * | *ACK|SYN * |*
>
> *| *25 * | *36050 * | *1547454657516057906 * | *-581795047 * | *20
> * | *ACK|RST * |*
>
> *| *31337 * | *36050 * | *2135088345829856782 * | *-581795047 * | *20
> * | *ACK|RST * |*
>
> *| *53 * | *36050 * | *6346604732028469374 * | *-581795047 * | *18
> * | *ACK|SYN * |*
>
> *| *113 * | *36061 * | *1324234686720796562 * | *-598572519 * | *20
> * | *ACK|RST * |*
>
> *| *80 * | *36050 * | *-9031405983396365775 * | *-581795047 * | *18
> * | *ACK|SYN * |*
>
> *| *70 * | *36050 * | *-3325173385040189964 * | *-581795047 * | *20
> * | *ACK|RST * |*
>
> *+-----------+-----------+-----------------------+-------------+------------+-------------------+*
>
> 10 rows selected (0.422 seconds)
>
> 0: jdbc:drill:zk=local> select src_port, dst_port, tcp_session, tcp_ack,
> tcp_flags, tcp_parsed_flags from
> dfs.root.`/Users/tdunning/tmp/synscan.pcap` where tcp_flags_rst>0 limit 10;
>
> *+-----------+-----------+-----------------------+-------------+------------+-------------------+*
>
> *| **src_port ** | **dst_port ** | ** tcp_session ** | ** tcp_ack
> ** | **tcp_flags ** | **tcp_parsed_flags ** |*
>
> *+-----------+-----------+-----------------------+-------------+------------+-------------------+*
>
> *| *113 * | *36050 * | *2591649313338334401 * | *-581795047 * | *20
> * | *ACK|RST * |*
>
> *| *25 * | *36050 * | *1547454657516057906 * | *-581795047 * | *20
> * | *ACK|RST * |*
>
> *| *31337 * | *36050 * | *2135088345829856782 * | *-581795047 * | *20
> * | *ACK|RST * |*
>
> *| *113 * | *36061 * | *1324234686720796562 * | *-598572519 * | *20
> * | *ACK|RST * |*
>
> *| *70 * | *36050 * | *-3325173385040189964 * | *-581795047 * | *20
> * | *ACK|RST * |*
>
> *+-----------+-----------+-----------------------+-------------+------------+-------------------+*
>
> 5 rows selected (0.278 seconds)
>
> On Tue, Jan 2, 2018 at 4:34 PM, Ted Dunning <te...@gmail.com> wrote:
>
>>
>> As predicted, those sequence and ack numbers were wrong.
>>
>> I now have a pull request that is much closer, in fact ready for review.
>>
>> See https://github.com/apache/drill/pull/1080
>>
>> *Charles, *
>>
>> Is this what you were looking to do?
>>
>> *Everybody,*
>>
>> On a side issue, there was a problem building this that required a change
>> to drill/exec/jdbc-all/pom.xml to increase the maximum allowable size for
>> the jdbc-all jar. That doesn't seem right, but it seemed to be a problem
>> even before I made a change.
>>
>>
>> On Tue, Jan 2, 2018 at 11:35 AM, Ted Dunning <te...@gmail.com>
>> wrote:
>>
>>>
>>> Charles,
>>>
>>> This got me excited enough to just code something up.
>>>
>>> Here is my current output:
>>>
>>> *| *-9112490598188246883 * | *120782885 * | *191 * | *NS|ECE (ECN
>>> capable)|URG|ACK|PSH|RST|SYN * |*
>>>
>>> *| *3418479551590665689 * | *120782885 * | *191 * | *NS|ECE (ECN
>>> capable)|URG|ACK|PSH|RST|SYN * |*
>>>
>>> *| *9404749046357206 * | *120782885 * | *191 * | *NS|ECE (ECN
>>> capable)|URG|ACK|PSH|RST|SYN * |*
>>>
>>> *| *7971558575805030122 * | *120782885 * | *191 * | *NS|ECE (ECN
>>> capable)|URG|ACK|PSH|RST|SYN * |*
>>>
>>> *| *-4666599101156419893 * | *120782885 * | *191 * | *NS|ECE (ECN
>>> capable)|URG|ACK|PSH|RST|SYN * |*
>>>
>>> *| *-672627876006511342 * | *-1846673370 * | *49 * | *ECE
>>> (Congestion experienced)|URG|SYN * |*
>>>
>>> *| *6346604732028469374 * | *1611798224 * | *106 * | *CWR|ECE
>>> (ECN capable)|ACK|RST * |*
>>>
>>> *| *-9031405983396365775 * | *1611798224 * | *201 * | *
>>> NS|CWR|ACK|SYN * |*
>>>
>>> *| *7738739733723725373 * | *1611798224 * | *219 * | *
>>> NS|CWR|URG|ACK|RST|SYN * |*
>>>
>>> *| *6346604732028469374 * | *1611798224 * | *106 * | *CWR|ECE
>>> (ECN capable)|ACK|RST * |*
>>>
>>> *| *-9031405983396365775 * | *1611798224 * | *201 * | *
>>> NS|CWR|ACK|SYN * |*
>>>
>>> *| *7738739733723725373 * | *1611798224 * | *219 * | *
>>> NS|CWR|URG|ACK|RST|SYN * |*
>>>
>>>
>>> I am not confident of the ack number being correct and will check against
>>> your reference. What is new here is the decoded flags column. It might be
>>> reasonable to have specific columns for the most important flags "ACK, RST,
>>> SYN" since all access is lazy anyway.
>>>
>>> What do you think?
>>>
>>> Can some or all of the pcap file you included be distributed as a unit
>>> test?
>>>
>>>
>>>
>>> On Mon, Jan 1, 2018 at 12:31 PM, Charles Givre <cg...@gmail.com> wrote:
>>>
>>>> Hello all,
>>>> I was playing with the PCAP functionality in Drill and I wanted to add
>>>> the TCP flags to the data that Drill is returning. I was also interested
>>>> in adding the TCP Sequence and Ack numbers as well. I noticed that the
>>>> code as written currently has a function in Packet.java which returns the
>>>> TCP Sequence number, however this was never added to the schema, so I added
>>>> that and rebuilt Drill, however, it doesn’t seem to be returning the
>>>> correct result. The file I was querying is attached to this email, and
>>>> should in all cases return a sequence number of zero.
>>>>
>>>> Questions:
>>>> 1. Could someone please take a look at the code for the tcp_sequence
>>>> and see if I did something wrong, or if the offset is not being calculated
>>>> correctly
>>>> 2. I’m trying to figure out the offsets for the various TCP flags. I
>>>> would think that the offset should be PacketConstants.ETHER_HEADER_LENGTH
>>>> + getIPHeaderLength() +13 to get the word that has the flags and then from
>>>> there, access the individual bits. However, this doesn’t seem to work.
>>>> What am I missing?
>>>> Thanks and Happy New Year!
>>>> - C
>>>
>>>
>>>
>>
Re: PCAP Issues
Posted by Ted Dunning <te...@gmail.com>.
Added exploded flag fields per Charles suggestion. I couldn't figure out
how to get a boolean value, however. Will push to the pull request shortly.
0: jdbc:drill:zk=local> select src_port, dst_port, tcp_session, tcp_ack,
tcp_flags, tcp_parsed_flags from
dfs.root.`/Users/tdunning/tmp/synscan.pcap` where tcp_flags_ack>0 limit 10;
*+-----------+-----------+-----------------------+-------------+------------+-------------------+*
*| **src_port ** | **dst_port ** | ** tcp_session ** | ** tcp_ack
** | **tcp_flags ** | **tcp_parsed_flags ** |*
*+-----------+-----------+-----------------------+-------------+------------+-------------------+*
*| *53 * | *36050 * | *6346604732028469374 * | *-581795047 * | *18
* | *ACK|SYN * |*
*| *113 * | *36050 * | *2591649313338334401 * | *-581795047 * | *20
* | *ACK|RST * |*
*| *80 * | *36050 * | *-9031405983396365775 * | *-581795047 * | *18
* | *ACK|SYN * |*
*| *22 * | *36050 * | *7738739733723725373 * | *-581795047 * | *18
* | *ACK|SYN * |*
*| *25 * | *36050 * | *1547454657516057906 * | *-581795047 * | *20
* | *ACK|RST * |*
*| *31337 * | *36050 * | *2135088345829856782 * | *-581795047 * | *20
* | *ACK|RST * |*
*| *53 * | *36050 * | *6346604732028469374 * | *-581795047 * | *18
* | *ACK|SYN * |*
*| *113 * | *36061 * | *1324234686720796562 * | *-598572519 * | *20
* | *ACK|RST * |*
*| *80 * | *36050 * | *-9031405983396365775 * | *-581795047 * | *18
* | *ACK|SYN * |*
*| *70 * | *36050 * | *-3325173385040189964 * | *-581795047 * | *20
* | *ACK|RST * |*
*+-----------+-----------+-----------------------+-------------+------------+-------------------+*
10 rows selected (0.422 seconds)
0: jdbc:drill:zk=local> select src_port, dst_port, tcp_session, tcp_ack,
tcp_flags, tcp_parsed_flags from
dfs.root.`/Users/tdunning/tmp/synscan.pcap` where tcp_flags_rst>0 limit 10;
*+-----------+-----------+-----------------------+-------------+------------+-------------------+*
*| **src_port ** | **dst_port ** | ** tcp_session ** | ** tcp_ack
** | **tcp_flags ** | **tcp_parsed_flags ** |*
*+-----------+-----------+-----------------------+-------------+------------+-------------------+*
*| *113 * | *36050 * | *2591649313338334401 * | *-581795047 * | *20
* | *ACK|RST * |*
*| *25 * | *36050 * | *1547454657516057906 * | *-581795047 * | *20
* | *ACK|RST * |*
*| *31337 * | *36050 * | *2135088345829856782 * | *-581795047 * | *20
* | *ACK|RST * |*
*| *113 * | *36061 * | *1324234686720796562 * | *-598572519 * | *20
* | *ACK|RST * |*
*| *70 * | *36050 * | *-3325173385040189964 * | *-581795047 * | *20
* | *ACK|RST * |*
*+-----------+-----------+-----------------------+-------------+------------+-------------------+*
5 rows selected (0.278 seconds)
On Tue, Jan 2, 2018 at 4:34 PM, Ted Dunning <te...@gmail.com> wrote:
>
> As predicted, those sequence and ack numbers were wrong.
>
> I now have a pull request that is much closer, in fact ready for review.
>
> See https://github.com/apache/drill/pull/1080
>
> *Charles, *
>
> Is this what you were looking to do?
>
> *Everybody,*
>
> On a side issue, there was a problem building this that required a change
> to drill/exec/jdbc-all/pom.xml to increase the maximum allowable size for
> the jdbc-all jar. That doesn't seem right, but it seemed to be a problem
> even before I made a change.
>
>
> On Tue, Jan 2, 2018 at 11:35 AM, Ted Dunning <te...@gmail.com>
> wrote:
>
>>
>> Charles,
>>
>> This got me excited enough to just code something up.
>>
>> Here is my current output:
>>
>> *| *-9112490598188246883 * | *120782885 * | *191 * | *NS|ECE (ECN
>> capable)|URG|ACK|PSH|RST|SYN * |*
>>
>> *| *3418479551590665689 * | *120782885 * | *191 * | *NS|ECE (ECN
>> capable)|URG|ACK|PSH|RST|SYN * |*
>>
>> *| *9404749046357206 * | *120782885 * | *191 * | *NS|ECE (ECN
>> capable)|URG|ACK|PSH|RST|SYN * |*
>>
>> *| *7971558575805030122 * | *120782885 * | *191 * | *NS|ECE (ECN
>> capable)|URG|ACK|PSH|RST|SYN * |*
>>
>> *| *-4666599101156419893 * | *120782885 * | *191 * | *NS|ECE (ECN
>> capable)|URG|ACK|PSH|RST|SYN * |*
>>
>> *| *-672627876006511342 * | *-1846673370 * | *49 * | *ECE
>> (Congestion experienced)|URG|SYN * |*
>>
>> *| *6346604732028469374 * | *1611798224 * | *106 * | *CWR|ECE
>> (ECN capable)|ACK|RST * |*
>>
>> *| *-9031405983396365775 * | *1611798224 * | *201 * | *
>> NS|CWR|ACK|SYN * |*
>>
>> *| *7738739733723725373 * | *1611798224 * | *219 * | *
>> NS|CWR|URG|ACK|RST|SYN * |*
>>
>> *| *6346604732028469374 * | *1611798224 * | *106 * | *CWR|ECE
>> (ECN capable)|ACK|RST * |*
>>
>> *| *-9031405983396365775 * | *1611798224 * | *201 * | *
>> NS|CWR|ACK|SYN * |*
>>
>> *| *7738739733723725373 * | *1611798224 * | *219 * | *
>> NS|CWR|URG|ACK|RST|SYN * |*
>>
>>
>> I am not confident of the ack number being correct and will check against
>> your reference. What is new here is the decoded flags column. It might be
>> reasonable to have specific columns for the most important flags "ACK, RST,
>> SYN" since all access is lazy anyway.
>>
>> What do you think?
>>
>> Can some or all of the pcap file you included be distributed as a unit
>> test?
>>
>>
>>
>> On Mon, Jan 1, 2018 at 12:31 PM, Charles Givre <cg...@gmail.com> wrote:
>>
>>> Hello all,
>>> I was playing with the PCAP functionality in Drill and I wanted to add
>>> the TCP flags to the data that Drill is returning. I was also interested
>>> in adding the TCP Sequence and Ack numbers as well. I noticed that the
>>> code as written currently has a function in Packet.java which returns the
>>> TCP Sequence number, however this was never added to the schema, so I added
>>> that and rebuilt Drill, however, it doesn’t seem to be returning the
>>> correct result. The file I was querying is attached to this email, and
>>> should in all cases return a sequence number of zero.
>>>
>>> Questions:
>>> 1. Could someone please take a look at the code for the tcp_sequence
>>> and see if I did something wrong, or if the offset is not being calculated
>>> correctly
>>> 2. I’m trying to figure out the offsets for the various TCP flags. I
>>> would think that the offset should be PacketConstants.ETHER_HEADER_LENGTH
>>> + getIPHeaderLength() +13 to get the word that has the flags and then from
>>> there, access the individual bits. However, this doesn’t seem to work.
>>> What am I missing?
>>> Thanks and Happy New Year!
>>> - C
>>
>>
>>
>
Re: PCAP Issues
Posted by Ted Dunning <te...@gmail.com>.
As predicted, those sequence and ack numbers were wrong.
I now have a pull request that is much closer, in fact ready for review.
See https://github.com/apache/drill/pull/1080
*Charles, *
Is this what you were looking to do?
*Everybody,*
On a side issue, there was a problem building this that required a change
to drill/exec/jdbc-all/pom.xml to increase the maximum allowable size for
the jdbc-all jar. That doesn't seem right, but it seemed to be a problem
even before I made a change.
On Tue, Jan 2, 2018 at 11:35 AM, Ted Dunning <te...@gmail.com> wrote:
>
> Charles,
>
> This got me excited enough to just code something up.
>
> Here is my current output:
>
> *| *-9112490598188246883 * | *120782885 * | *191 * | *NS|ECE (ECN
> capable)|URG|ACK|PSH|RST|SYN * |*
>
> *| *3418479551590665689 * | *120782885 * | *191 * | *NS|ECE (ECN
> capable)|URG|ACK|PSH|RST|SYN * |*
>
> *| *9404749046357206 * | *120782885 * | *191 * | *NS|ECE (ECN
> capable)|URG|ACK|PSH|RST|SYN * |*
>
> *| *7971558575805030122 * | *120782885 * | *191 * | *NS|ECE (ECN
> capable)|URG|ACK|PSH|RST|SYN * |*
>
> *| *-4666599101156419893 * | *120782885 * | *191 * | *NS|ECE (ECN
> capable)|URG|ACK|PSH|RST|SYN * |*
>
> *| *-672627876006511342 * | *-1846673370 * | *49 * | *ECE
> (Congestion experienced)|URG|SYN * |*
>
> *| *6346604732028469374 * | *1611798224 * | *106 * | *CWR|ECE (ECN
> capable)|ACK|RST * |*
>
> *| *-9031405983396365775 * | *1611798224 * | *201 * | *
> NS|CWR|ACK|SYN * |*
>
> *| *7738739733723725373 * | *1611798224 * | *219 * | *
> NS|CWR|URG|ACK|RST|SYN * |*
>
> *| *6346604732028469374 * | *1611798224 * | *106 * | *CWR|ECE (ECN
> capable)|ACK|RST * |*
>
> *| *-9031405983396365775 * | *1611798224 * | *201 * | *
> NS|CWR|ACK|SYN * |*
>
> *| *7738739733723725373 * | *1611798224 * | *219 * | *
> NS|CWR|URG|ACK|RST|SYN * |*
>
>
> I am not confident of the ack number being correct and will check against
> your reference. What is new here is the decoded flags column. It might be
> reasonable to have specific columns for the most important flags "ACK, RST,
> SYN" since all access is lazy anyway.
>
> What do you think?
>
> Can some or all of the pcap file you included be distributed as a unit
> test?
>
>
>
> On Mon, Jan 1, 2018 at 12:31 PM, Charles Givre <cg...@gmail.com> wrote:
>
>> Hello all,
>> I was playing with the PCAP functionality in Drill and I wanted to add
>> the TCP flags to the data that Drill is returning. I was also interested
>> in adding the TCP Sequence and Ack numbers as well. I noticed that the
>> code as written currently has a function in Packet.java which returns the
>> TCP Sequence number, however this was never added to the schema, so I added
>> that and rebuilt Drill, however, it doesn’t seem to be returning the
>> correct result. The file I was querying is attached to this email, and
>> should in all cases return a sequence number of zero.
>>
>> Questions:
>> 1. Could someone please take a look at the code for the tcp_sequence and
>> see if I did something wrong, or if the offset is not being calculated
>> correctly
>> 2. I’m trying to figure out the offsets for the various TCP flags. I
>> would think that the offset should be PacketConstants.ETHER_HEADER_LENGTH
>> + getIPHeaderLength() +13 to get the word that has the flags and then from
>> there, access the individual bits. However, this doesn’t seem to work.
>> What am I missing?
>> Thanks and Happy New Year!
>> - C
>
>
>
Re: PCAP Issues
Posted by Ted Dunning <te...@gmail.com>.
Charles,
This got me excited enough to just code something up.
Here is my current output:
*| *-9112490598188246883 * | *120782885 * | *191 * | *NS|ECE (ECN
capable)|URG|ACK|PSH|RST|SYN * |*
*| *3418479551590665689 * | *120782885 * | *191 * | *NS|ECE (ECN
capable)|URG|ACK|PSH|RST|SYN * |*
*| *9404749046357206 * | *120782885 * | *191 * | *NS|ECE (ECN
capable)|URG|ACK|PSH|RST|SYN * |*
*| *7971558575805030122 * | *120782885 * | *191 * | *NS|ECE (ECN
capable)|URG|ACK|PSH|RST|SYN * |*
*| *-4666599101156419893 * | *120782885 * | *191 * | *NS|ECE (ECN
capable)|URG|ACK|PSH|RST|SYN * |*
*| *-672627876006511342 * | *-1846673370 * | *49 * | *ECE
(Congestion experienced)|URG|SYN * |*
*| *6346604732028469374 * | *1611798224 * | *106 * | *CWR|ECE (ECN
capable)|ACK|RST * |*
*| *-9031405983396365775 * | *1611798224 * | *201 * | *NS|CWR|ACK|SYN
* |*
*| *7738739733723725373 * | *1611798224 * | *219 * | *
NS|CWR|URG|ACK|RST|SYN * |*
*| *6346604732028469374 * | *1611798224 * | *106 * | *CWR|ECE (ECN
capable)|ACK|RST * |*
*| *-9031405983396365775 * | *1611798224 * | *201 * | *NS|CWR|ACK|SYN
* |*
*| *7738739733723725373 * | *1611798224 * | *219 * | *
NS|CWR|URG|ACK|RST|SYN * |*
I am not confident of the ack number being correct and will check against
your reference. What is new here is the decoded flags column. It might be
reasonable to have specific columns for the most important flags "ACK, RST,
SYN" since all access is lazy anyway.
What do you think?
Can some or all of the pcap file you included be distributed as a unit test?
On Mon, Jan 1, 2018 at 12:31 PM, Charles Givre <cg...@gmail.com> wrote:
> Hello all,
> I was playing with the PCAP functionality in Drill and I wanted to add the
> TCP flags to the data that Drill is returning. I was also interested in
> adding the TCP Sequence and Ack numbers as well. I noticed that the code
> as written currently has a function in Packet.java which returns the TCP
> Sequence number, however this was never added to the schema, so I added
> that and rebuilt Drill, however, it doesn’t seem to be returning the
> correct result. The file I was querying is attached to this email, and
> should in all cases return a sequence number of zero.
>
> Questions:
> 1. Could someone please take a look at the code for the tcp_sequence and
> see if I did something wrong, or if the offset is not being calculated
> correctly
> 2. I’m trying to figure out the offsets for the various TCP flags. I
> would think that the offset should be PacketConstants.ETHER_HEADER_LENGTH
> + getIPHeaderLength() +13 to get the word that has the flags and then from
> there, access the individual bits. However, this doesn’t seem to work.
> What am I missing?
> Thanks and Happy New Year!
> - C
Re: PCAP Issues
Posted by Ted Dunning <te...@gmail.com>.
Attachments are stripped.
Can you provide a URL for this file somewhere?
On Mon, Jan 1, 2018 at 12:31 PM, Charles Givre <cg...@gmail.com> wrote:
> Hello all,
> I was playing with the PCAP functionality in Drill and I wanted to add the
> TCP flags to the data that Drill is returning. I was also interested in
> adding the TCP Sequence and Ack numbers as well. I noticed that the code
> as written currently has a function in Packet.java which returns the TCP
> Sequence number, however this was never added to the schema, so I added
> that and rebuilt Drill, however, it doesn’t seem to be returning the
> correct result. The file I was querying is attached to this email, and
> should in all cases return a sequence number of zero.
>
> Questions:
> 1. Could someone please take a look at the code for the tcp_sequence and
> see if I did something wrong, or if the offset is not being calculated
> correctly
> 2. I’m trying to figure out the offsets for the various TCP flags. I
> would think that the offset should be PacketConstants.ETHER_HEADER_LENGTH
> + getIPHeaderLength() +13 to get the word that has the flags and then from
> there, access the individual bits. However, this doesn’t seem to work.
> What am I missing?
> Thanks and Happy New Year!
> - C