You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Terry Kennedy <TE...@tmk.com> on 2011/07/28 09:20:40 UTC
[users@httpd] LogFormat Combined - many logfile lines with no Referer or User-agent
I'm using the default "LogFormat combined" directive in my httpd.conf
file. That should generate logfile lines using this pattern:
"%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""
There have always been occasional entries which don't contain the last
2 fields, for some reason.
However, I have observed a HUGE increase in the number of logfile lines
missing these two fields, starting early in June, 2011. As an example,
one server (with a relatively constant number of requests over time) had
been logging 2,000 to 10,000 of these lines with missing data from August
2010 through May 2011. The number for June, 2011 was 496,700 lines.
Even if the Referer and User-agent data was missing for some reason, I
would have expected httpd to log lines ending in "" "" or possibly "-" "-",
since there are escaped literal quotes on either side of both of those
fields in the LogFormat config line. This makes me think that it is some-
thing going on inside Apache (perhaps triggered by some external change).
There doesn't seem to be any pattern to client IP address, browser, etc.
This is causing problems with Analog, since it discards any lines that
have a missing Referer field, and thus won't report those requests.
My environment has a number of systems that are experiencing this issue.
However, other system seem mostly unaffected, with the number of these
lines below 10000 every month. The numbers above are from a system running
FreeBSD 6.4 and Apache 2.0.63 (I know, it's in the process of being up-
graded). That system has no virtual hosts and a relatively simple config.
I'm also seeing this on a newly-installed FreeBSD 8.2 system with Apache
2.2.19, though I don't have any historical data for this system as it is a
new install.
I can't reproduce whatever events are causing this, either - a request
that was logged with a missing Referer and User-Agent shows up properly
when I try it again from the same client.
Any ideas as to what I should do to troubleshoot this further?
Terry Kennedy http://www.tmk.com
terry@tmk.com New York, NY USA
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] LogFormat Combined - many logfile lines with
no Referer or User-agent
Posted by Stormy <st...@stormy.ca>.
At 09:40 AM 7/28/2011 -0400, Rich Bowen wrote:
[snip]
> > However, I have observed a HUGE increase in the number of logfile lines
> > missing these two fields, starting early in June, 2011.
>
>It would be interesting to see what version of what browser released in
>the last 30 days.
FireFox 5 ... ???
Paul
Tired old sys-admin
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] LogFormat Combined - many logfile lines with no Referer or User-agent
Posted by Rich Bowen <rb...@rcbowen.com>.
On Jul 28, 2011, at 3:20 AM, Terry Kennedy wrote:
> I'm using the default "LogFormat combined" directive in my httpd.conf
> file. That should generate logfile lines using this pattern:
>
> "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""
>
> There have always been occasional entries which don't contain the last
> 2 fields, for some reason.
These are optional fields which *may* be passed by a user agent. When they are passed, they are not reliable - that is, they may be spoofed, trivially.
>
> However, I have observed a HUGE increase in the number of logfile lines
> missing these two fields, starting early in June, 2011.
It would be interesting to see what version of what browser released in the last 30 days.
> Even if the Referer and User-agent data was missing for some reason, I
> would have expected httpd to log lines ending in "" "" or possibly "-" "-",
> since there are escaped literal quotes on either side of both of those
> fields in the LogFormat config line. This makes me think that it is some-
> thing going on inside Apache (perhaps triggered by some external change).
Oh. Hmm. That's interesting. What I would look for, in that case, is more than one LogFormat directive logging to the same location.
--
Rich Bowen
rbowen@rcbowen.com
rbowen@apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] LogFormat Combined - many logfile lines with no
Referer or User-agent
Posted by Marcin 'Rambo' Roguski <ra...@id.uw.edu.pl>.
> There doesn't seem to be any pattern to client IP address, browser, etc.
I know for a fact, that certain browsers (most versions of IE for example), don't send
referer when request is induced via JavaScript. Several firewalls strip these by default, too.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org