You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Apache Issues <ap...@yahoo.com> on 2010/08/31 01:00:34 UTC
[users@httpd] What IP address is this log entry coming from? (Is "::" a valid IP address?)
I'm using:
CustomLog "/var/log/apache2/access_log" "%a %l %u %t \"%r\" %>s %b
\"%{Referer}i\""
And I occasionally see this right around the time the CPU starts running at
100%:
:: - - [27/Aug/2010:12:28:01 -0700] "GET /favicon.ico HTTP/1.1" 200 - "-"
%a is supposed to be an IP address, so what IP address is "::"? I'm only
somewhat familiar with IPv6 but I've never seen "::" before.
I'm attempting to try to figure out why, on random occasions, the server CPU
suddenly spikes to 100%. I'm not sure where the problem lies. Could be Apache,
MySQL, PHP, or something else (e.g. the OS). This problem has been ongoing for
months now and the only "fix" is to reboot the box, which is a rather
frustrating "solution". The "::" issue is the first possible clue I've gotten.
It seems to crop up only around the times the server is at 100% in the logs.
Re: [users@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
Posted by Eric Covener <co...@gmail.com>.
>> That is the source IP address, which is required for routing replies
>> (including those during the "connect" flow) back to the client, so I don't
>> see how this can be 0 simply because of something the client is doing
>> (other
>> than triggering some sort of bug on the server side, of course).
>>
> scanning for an open proxy, the :: would grab the ip address and the proxy
> would then be in it's use to route content that may not be legitimately sent
> to the destination. or content that it is criminal to even possess. [ child
> porn jumps to mind for an example. ]
> I suspect that the client is a bot doing a scan for an open proxy, one
> handing the :: out for that purpose. or, since it causes the 100% cpu load,
> it's the beginnings of a new ddos attack mechanism. we all may need to
> explicitly deny unspecified ip addresses from server access right quick.
Do you have any theories about the moon landing?
--
Eric Covener
covener@gmail.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What IP address is this log entry coming from?
(Is "::" a valid IP address?)
Posted by "J. Greenlees" <li...@jaqui-greenlees.net>.
Jeff Trawick wrote:
> On Thu, Sep 2, 2010 at 5:40 PM, J. Greenlees <li...@jaqui-greenlees.net>wrote:
>
>> Jeff Trawick wrote:
>> ~snip~
>>
>> %a is supposed to be an IP address, so what IP address is "::"? I'm only
>>>>> somewhat familiar with IPv6 but I've never seen "::" before.
>>>>>
>>>> http://en.wikipedia.org/wiki/IPv6_address#Notation
>>>>
>>>> One or any number of consecutive groups of zero value may be replaced
>>>> with two colons. [ ... ]
>>>>
>>>> The localhost (loopback) address, 0:0:0:0:0:0:0:1, and the IPv6
>>>> unspecified address, 0:0:0:0:0:0:0:0, are reduced to ::1 and ::,
>>>> respectively.
>>>>
>>>>
>>> and it is bogus to have the unspecified address as the client IP address
>>>
>> and if you check MS' RPC mechanism it uses 0.0.0.0 for the ip address to
>> glom onto ANY available ip address. That suggests that the client giving the
>> :: address is most likely a bot of some sort.
>> it could be a legitimate bot for an rpc mechanism, or it could be [ seems
>> more likely ] to be one meant to find an exploitable weakness.
>>
>> or, the client could be using an anonymizer service before getting to the
>> OPs site.
>>
>> many reasons that it could be the ip unspecified address, only a few of
>> which are cause for concern to the server admin.
>
>
> That is the source IP address, which is required for routing replies
> (including those during the "connect" flow) back to the client, so I don't
> see how this can be 0 simply because of something the client is doing (other
> than triggering some sort of bug on the server side, of course).
>
scanning for an open proxy, the :: would grab the ip address and the
proxy would then be in it's use to route content that may not be
legitimately sent to the destination. or content that it is criminal to
even possess. [ child porn jumps to mind for an example. ]
I suspect that the client is a bot doing a scan for an open proxy, one
handing the :: out for that purpose. or, since it causes the 100% cpu
load, it's the beginnings of a new ddos attack mechanism. we all may
need to explicitly deny unspecified ip addresses from server access
right quick.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
Posted by Jeff Trawick <tr...@gmail.com>.
On Thu, Sep 2, 2010 at 5:40 PM, J. Greenlees <li...@jaqui-greenlees.net>wrote:
> Jeff Trawick wrote:
> ~snip~
>
> %a is supposed to be an IP address, so what IP address is "::"? I'm only
>>>> somewhat familiar with IPv6 but I've never seen "::" before.
>>>>
>>> http://en.wikipedia.org/wiki/IPv6_address#Notation
>>>
>>> One or any number of consecutive groups of zero value may be replaced
>>> with two colons. [ ... ]
>>>
>>> The localhost (loopback) address, 0:0:0:0:0:0:0:1, and the IPv6
>>> unspecified address, 0:0:0:0:0:0:0:0, are reduced to ::1 and ::,
>>> respectively.
>>>
>>>
>> and it is bogus to have the unspecified address as the client IP address
>>
>
> and if you check MS' RPC mechanism it uses 0.0.0.0 for the ip address to
> glom onto ANY available ip address. That suggests that the client giving the
> :: address is most likely a bot of some sort.
> it could be a legitimate bot for an rpc mechanism, or it could be [ seems
> more likely ] to be one meant to find an exploitable weakness.
>
> or, the client could be using an anonymizer service before getting to the
> OPs site.
>
> many reasons that it could be the ip unspecified address, only a few of
> which are cause for concern to the server admin.
That is the source IP address, which is required for routing replies
(including those during the "connect" flow) back to the client, so I don't
see how this can be 0 simply because of something the client is doing (other
than triggering some sort of bug on the server side, of course).
Re: [users@httpd] What IP address is this log entry coming from?
(Is "::" a valid IP address?)
Posted by "J. Greenlees" <li...@jaqui-greenlees.net>.
Jeff Trawick wrote:
~snip~
>>> %a is supposed to be an IP address, so what IP address is "::"? I'm only
>>> somewhat familiar with IPv6 but I've never seen "::" before.
>> http://en.wikipedia.org/wiki/IPv6_address#Notation
>>
>> One or any number of consecutive groups of zero value may be replaced
>> with two colons. [ ... ]
>>
>> The localhost (loopback) address, 0:0:0:0:0:0:0:1, and the IPv6
>> unspecified address, 0:0:0:0:0:0:0:0, are reduced to ::1 and ::,
>> respectively.
>>
>
> and it is bogus to have the unspecified address as the client IP address
and if you check MS' RPC mechanism it uses 0.0.0.0 for the ip address to
glom onto ANY available ip address. That suggests that the client giving
the :: address is most likely a bot of some sort.
it could be a legitimate bot for an rpc mechanism, or it could be [
seems more likely ] to be one meant to find an exploitable weakness.
or, the client could be using an anonymizer service before getting to
the OPs site.
many reasons that it could be the ip unspecified address, only a few of
which are cause for concern to the server admin.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
Posted by Jeff Trawick <tr...@gmail.com>.
On Wed, Sep 1, 2010 at 11:12 AM, Tom Evans <te...@googlemail.com> wrote:
> On Tue, Aug 31, 2010 at 12:00 AM, Apache Issues <ap...@yahoo.com>
> wrote:
> > I'm using:
> >
> > CustomLog "/var/log/apache2/access_log" "%a %l %u %t \"%r\" %>s %b
> > \"%{Referer}i\""
> >
> > And I occasionally see this right around the time the CPU starts running
> at
> > 100%:
> >
> > :: - - [27/Aug/2010:12:28:01 -0700] "GET /favicon.ico HTTP/1.1" 200 - "-"
> >
> > %a is supposed to be an IP address, so what IP address is "::"? I'm only
> > somewhat familiar with IPv6 but I've never seen "::" before.
>
> http://en.wikipedia.org/wiki/IPv6_address#Notation
>
> One or any number of consecutive groups of zero value may be replaced
> with two colons. [ ... ]
>
> The localhost (loopback) address, 0:0:0:0:0:0:0:1, and the IPv6
> unspecified address, 0:0:0:0:0:0:0:0, are reduced to ::1 and ::,
> respectively.
>
and it is bogus to have the unspecified address as the client IP address
Re: [users@httpd] What IP address is this log entry coming from? (Is "::" a valid IP address?)
Posted by Apache Issues <ap...@yahoo.com>.
Turns out the box only had 2GB RAM and was simply running out of physical memory
- running a web server on swap is a Bad Idea(TM). We tossed in 8GB more (10GB
RAM total) and the problem seems to have gone away. It was a real
head-scratcher since we were told that the box had "awesome hardware" so the
thought that there might not be enough hardware never even crossed our minds.
The "make sure MaxClients isn't way over the capacity of your machine" tip below
is what led us to the 2GB RAM issue. Thanks Jeff. You're the man!
________________________________
From: Apache Issues <ap...@yahoo.com>
To: users@httpd.apache.org
Sent: Thu, September 2, 2010 1:50:10 PM
Subject: Re: [users@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
We're trying a few of your suggestions. Kind of hard to test it. Thanks for
the quick responses.
________________________________
From: Jeff Trawick <tr...@gmail.com>
To: users@httpd.apache.org
Sent: Thu, September 2, 2010 12:34:43 PM
Subject: Re: [users@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
On Thu, Sep 2, 2010 at 3:03 PM, Apache Issues <ap...@yahoo.com> wrote:
That won't work. I can't even get results from 'ps aux' to get a pid, SSH is
super laggy (I can type maybe one character every 4 or 5 seconds - if I'm that
lucky), and most commands never complete.
>
>The hardware has been tested and is fine. The problem occurs randomly - we've
>gone a week without incident before but we've also had days where this problem
>crops up 4-6 times throughout the day. The "::" log entries are the first real
>clue to the cause. And the only "solution" we've come up with is to reboot the
>box.
>
wild ideas
any monitoring scripts which can take action at the first sign of the problem
might be more successful than interactive attempts at capturing ps, running
gcore against high CPU process, whatever
change your "Listen portnumber" directives to "Listen 0.0.0.0:portnumber" to
avoid httpd seeing an IPv6 connection (especially one with no source address or
which it somehow mangles to look like that)
CPU rlimits on httpd perhaps? (I don't think I've ever tried that)
Can you nice the httpd down? (haven't tried that)
make sure MaxClients isn't way over the capacity of your machine
Re: [users@httpd] What IP address is this log entry coming from? (Is "::" a valid IP address?)
Posted by Apache Issues <ap...@yahoo.com>.
We're trying a few of your suggestions. Kind of hard to test it. Thanks for
the quick responses.
________________________________
From: Jeff Trawick <tr...@gmail.com>
To: users@httpd.apache.org
Sent: Thu, September 2, 2010 12:34:43 PM
Subject: Re: [users@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
On Thu, Sep 2, 2010 at 3:03 PM, Apache Issues <ap...@yahoo.com> wrote:
That won't work. I can't even get results from 'ps aux' to get a pid, SSH is
super laggy (I can type maybe one character every 4 or 5 seconds - if I'm that
lucky), and most commands never complete.
>
>The hardware has been tested and is fine. The problem occurs randomly - we've
>gone a week without incident before but we've also had days where this problem
>crops up 4-6 times throughout the day. The "::" log entries are the first real
>clue to the cause. And the only "solution" we've come up with is to reboot the
>box.
>
wild ideas
any monitoring scripts which can take action at the first sign of the problem
might be more successful than interactive attempts at capturing ps, running
gcore against high CPU process, whatever
change your "Listen portnumber" directives to "Listen 0.0.0.0:portnumber" to
avoid httpd seeing an IPv6 connection (especially one with no source address or
which it somehow mangles to look like that)
CPU rlimits on httpd perhaps? (I don't think I've ever tried that)
Can you nice the httpd down? (haven't tried that)
make sure MaxClients isn't way over the capacity of your machine
Re: [users@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
Posted by Jeff Trawick <tr...@gmail.com>.
On Thu, Sep 2, 2010 at 3:03 PM, Apache Issues <ap...@yahoo.com>wrote:
> That won't work. I can't even get results from 'ps aux' to get a pid, SSH
> is super laggy (I can type maybe one character every 4 or 5 seconds - if I'm
> that lucky), and most commands never complete.
>
> The hardware has been tested and is fine. The problem occurs randomly -
> we've gone a week without incident before but we've also had days where this
> problem crops up 4-6 times throughout the day. The "::" log entries are the
> first real clue to the cause. And the only "solution" we've come up with is
> to reboot the box.
>
wild ideas
any monitoring scripts which can take action at the first sign of the
problem might be more successful than interactive attempts at capturing ps,
running gcore against high CPU process, whatever
change your "Listen portnumber" directives to "Listen 0.0.0.0:portnumber" to
avoid httpd seeing an IPv6 connection (especially one with no source address
or which it somehow mangles to look like that)
CPU rlimits on httpd perhaps? (I don't think I've ever tried that)
Can you nice the httpd down? (haven't tried that)
make sure MaxClients isn't way over the capacity of your machine
Re: [users@httpd] What IP address is this log entry coming from? (Is "::" a valid IP address?)
Posted by Apache Issues <ap...@yahoo.com>.
That won't work. I can't even get results from 'ps aux' to get a pid, SSH is
super laggy (I can type maybe one character every 4 or 5 seconds - if I'm that
lucky), and most commands never complete.
The hardware has been tested and is fine. The problem occurs randomly - we've
gone a week without incident before but we've also had days where this problem
crops up 4-6 times throughout the day. The "::" log entries are the first real
clue to the cause. And the only "solution" we've come up with is to reboot the
box.
________________________________
From: Jeff Trawick <tr...@gmail.com>
To: users@httpd.apache.org
Sent: Thu, September 2, 2010 11:32:10 AM
Subject: Re: [users@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
On Thu, Sep 2, 2010 at 2:24 PM, Apache Issues <ap...@yahoo.com> wrote:
Our server just went nuts again. And again "::" shows up in the logs right
around the moment it started chugging 100% CPU. Help!
>
attach to the high CPU httpd process with a debugger and get backtraces
see http://httpd.apache.org/dev/debugging.html
Re: [users@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
Posted by Jeff Trawick <tr...@gmail.com>.
On Thu, Sep 2, 2010 at 2:24 PM, Apache Issues <ap...@yahoo.com>wrote:
> Our server just went nuts again. And again "::" shows up in the logs right
> around the moment it started chugging 100% CPU. Help!
>
attach to the high CPU httpd process with a debugger and get backtraces
see http://httpd.apache.org/dev/debugging.html
Re: [users@httpd] What IP address is this log entry coming from? (Is "::" a valid IP address?)
Posted by Apache Issues <ap...@yahoo.com>.
Our server just went nuts again. And again "::" shows up in the logs right
around the moment it started chugging 100% CPU. Help!
________________________________
From: Apache Issues <ap...@yahoo.com>
To: users@httpd.apache.org
Sent: Wed, September 1, 2010 11:36:02 AM
Subject: Re: [users@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
Oooookay... That makes it rather difficult to track down a solution. Any
theory as to why this occurs/how this can occur would be quite helpful.
________________________________
From: Tom Evans <te...@googlemail.com>
To: users@httpd.apache.org
Sent: Wed, September 1, 2010 8:12:23 AM
Subject: Re: [users@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
On Tue, Aug 31, 2010 at 12:00 AM, Apache Issues <ap...@yahoo.com> wrote:
> I'm using:
>
> CustomLog "/var/log/apache2/access_log" "%a %l %u %t \"%r\" %>s %b
> \"%{Referer}i\""
>
> And I occasionally see this right around the time the CPU starts running at
> 100%:
>
> :: - - [27/Aug/2010:12:28:01 -0700] "GET /favicon.ico HTTP/1.1" 200 - "-"
>
> %a is supposed to be an IP address, so what IP address is "::"? I'm only
> somewhat familiar with IPv6 but I've never seen "::" before.
http://en.wikipedia.org/wiki/IPv6_address#Notation
One or any number of consecutive groups of zero value may be replaced
with two colons. [ ... ]
The localhost (loopback) address, 0:0:0:0:0:0:0:1, and the IPv6
unspecified address, 0:0:0:0:0:0:0:0, are reduced to ::1 and ::,
respectively.
Cheers
Tom
Re: [users@httpd] What IP address is this log entry coming from? (Is "::" a valid IP address?)
Posted by Apache Issues <ap...@yahoo.com>.
Oooookay... That makes it rather difficult to track down a solution. Any
theory as to why this occurs/how this can occur would be quite helpful.
________________________________
From: Tom Evans <te...@googlemail.com>
To: users@httpd.apache.org
Sent: Wed, September 1, 2010 8:12:23 AM
Subject: Re: [users@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
On Tue, Aug 31, 2010 at 12:00 AM, Apache Issues <ap...@yahoo.com> wrote:
> I'm using:
>
> CustomLog "/var/log/apache2/access_log" "%a %l %u %t \"%r\" %>s %b
> \"%{Referer}i\""
>
> And I occasionally see this right around the time the CPU starts running at
> 100%:
>
> :: - - [27/Aug/2010:12:28:01 -0700] "GET /favicon.ico HTTP/1.1" 200 - "-"
>
> %a is supposed to be an IP address, so what IP address is "::"? I'm only
> somewhat familiar with IPv6 but I've never seen "::" before.
http://en.wikipedia.org/wiki/IPv6_address#Notation
One or any number of consecutive groups of zero value may be replaced
with two colons. [ ... ]
The localhost (loopback) address, 0:0:0:0:0:0:0:1, and the IPv6
unspecified address, 0:0:0:0:0:0:0:0, are reduced to ::1 and ::,
respectively.
Cheers
Tom
Re: [users@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
Posted by Tom Evans <te...@googlemail.com>.
On Tue, Aug 31, 2010 at 12:00 AM, Apache Issues <ap...@yahoo.com> wrote:
> I'm using:
>
> CustomLog "/var/log/apache2/access_log" "%a %l %u %t \"%r\" %>s %b
> \"%{Referer}i\""
>
> And I occasionally see this right around the time the CPU starts running at
> 100%:
>
> :: - - [27/Aug/2010:12:28:01 -0700] "GET /favicon.ico HTTP/1.1" 200 - "-"
>
> %a is supposed to be an IP address, so what IP address is "::"? I'm only
> somewhat familiar with IPv6 but I've never seen "::" before.
http://en.wikipedia.org/wiki/IPv6_address#Notation
One or any number of consecutive groups of zero value may be replaced
with two colons. [ ... ]
The localhost (loopback) address, 0:0:0:0:0:0:0:1, and the IPv6
unspecified address, 0:0:0:0:0:0:0:0, are reduced to ::1 and ::,
respectively.
Cheers
Tom
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org