You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Eric Frazier <ef...@kwinternet.com> on 2003/09/05 02:49:50 UTC
[users@httpd] What is unique? Access stats and IP's
Hi,
I just wanted to get an opinon. When you consider the idea of a unique
vistor, the brain dead way of doing that is to say, one IP one visitor, or
one cookie one visitor. But aside from cookies, when using IP address, you
do have to take into account that a vistior from an IP address at 8AM may
very well not be the same vistor as at 8PM because they are both behind the
same firewall/proxy. So one option is to consider that every unique ip
address within an hour is considered to be a unique vistor. But that seems
off as well because then if you compare overall stats and compare the number
of unique vistors by sorting by unique IP daily, vs the number of unique
vistors by sorting the IP hourly, the daily number can be like 3 times more
than the hourly. So now I am starting to wonder, are we really just left to
using cookies for this? What bugs me about that is the stats I see from our
sites for Javascript turned off in browsers is actualy pretty high, so if JS
is off, I would not at all be shocked to see cookies being rejected as well.
Maybe the only way is a hybrid of if the cookie was rejected then consider
the ip, but if there was a cookie for that connection then ignore any IP data.
If any of the above makes sense to you, and you have an opinon about it I
would love to hear it.
Thanks,
Eric
(250) 655 - 9513 (PST Time Zone)
"Inquiry is fatal to certainty." -- Will Durant
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is unique? Access stats and IP's
Posted by Bill Moran <wm...@potentialtech.com>.
Joshua Slive wrote:
> On Fri, 5 Sep 2003, Bill Moran wrote:
<snip>
>>The problem is rooted in the fact that HTTP was not really intended for
>>security, and therefore not really intended for logging.
>
> Huh? Where do you get that from?
Well, perhaps I worded my introduction poorly, but if you read the rest
of my response you'll see we agreed on just about every other point.
--
Bill Moran
Potential Technologies
http://www.potentialtech.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is unique? Access stats and IP's
Posted by Joshua Slive <jo...@slive.ca>.
On Fri, 5 Sep 2003, Bill Moran wrote:
> Eric Frazier wrote:
> > Hi,
> >
> > I just wanted to get an opinon. When you consider the idea of a unique
> > vistor, the brain dead way of doing that is to say, one IP one visitor, or
> > one cookie one visitor. But aside from cookies, when using IP address, you
> > do have to take into account that a vistior from an IP address at 8AM may
> > very well not be the same vistor as at 8PM because they are both behind the
> > same firewall/proxy.
It's much worse than that. Some big sites like AOL use a rotating series
of proxies, so you can easily have multiple hits from the same IP address
within a minute all coming from different people, AND you can have
multiple hits from different IP addresses within a minute all coming from
the same person. Plus unless you use cache-busting techniques, many of
your "sessions" will have their beginning or ending cut off.
> The problem is rooted in the fact that HTTP was not really intended for
> security, and therefore not really intended for logging.
Huh? Where do you get that from?
The problem is that HTTP is a stateless protocol. Each HTTP request looks
completely independent. This is a major benefit of the protocol in many
different ways, ranging from security to cachability, to the usefulness of
the URL as a reference to an object.
But it obviously makes life more difficult for people who want to keep
state. The obvious solution is cookies, which are designed exactly to
bridge the state gap. Unfortunately, somewhat of a hysteria has developed
over cookies. Session cookies (those that are erased when the browser is
closed) have no real privacy implications, and there is no reason to block
them. But many people do anyway. (Cookies do have their disadvantages
for the website operator, however. For example, they break cachability.)
So what is the solution? There is no easy one. If you really need to
track sessions and individual visitors, you can use some combination of
cookies, IP address, URL-path-cookie, referer, and time. But you'll never
get something perfectly exact. You'll only get an estimate.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is unique? Access stats and IP's
Posted by Bill Moran <wm...@potentialtech.com>.
Eric Frazier wrote:
> Hi,
>
> I just wanted to get an opinon. When you consider the idea of a unique
> vistor, the brain dead way of doing that is to say, one IP one visitor, or
> one cookie one visitor. But aside from cookies, when using IP address, you
> do have to take into account that a vistior from an IP address at 8AM may
> very well not be the same vistor as at 8PM because they are both behind the
> same firewall/proxy. So one option is to consider that every unique ip
> address within an hour is considered to be a unique vistor. But that seems
> off as well because then if you compare overall stats and compare the number
> of unique vistors by sorting by unique IP daily, vs the number of unique
> vistors by sorting the IP hourly, the daily number can be like 3 times more
> than the hourly. So now I am starting to wonder, are we really just left to
> using cookies for this? What bugs me about that is the stats I see from our
> sites for Javascript turned off in browsers is actualy pretty high, so if JS
> is off, I would not at all be shocked to see cookies being rejected as well.
>
> Maybe the only way is a hybrid of if the cookie was rejected then consider
> the ip, but if there was a cookie for that connection then ignore any IP data.
>
> If any of the above makes sense to you, and you have an opinon about it I
> would love to hear it.
This is getting into the theoretical/hypothetical/existential.
It's so bad, I once had a professional statistician tell me that "hits" and
other stats generated by web servers were useless.
The problem is rooted in the fact that HTTP was not really intended for
security, and therefore not really intended for logging. Couple this with
the fact that most administrators are interested in how much load is on
their server (and thus "is it powerful enough to keep up") rather than how
many people are generating that load.
The problem is, there are NO "honest" statistics available for guaging how
many people have viewed your web site. Even "hits" is inaccurate, because
it doen't take into account the situation where one person is sitting at
the computer and two more are watching over his shoulder. Obviously, it
can't do that.
This is why you see so many sites requiring you to "register" before you
get to see the really good stuff. It's not so much an attempt to invade
your privacy, as it is an attempt to honestly ascertain how many people
are coming to the site. Unfortunately, this isn't even accurate because
most people don't take it seriously, and people _will_ share passwords.
Most web statistical programs will use a time frame (such as 30 minutes)
and generate an estimate on the number of "visits" based on the technique
you describe above. This is skewed by nat proxies (that may give 100s of
computers the same address) and public computers that may have many people
use them in the course of one time period.
The fact is, any statistic you try to use to determine the number of
people visiting your site is an approximation. Period. Just keep that
in mind when you analyze them. If you MUST have those estimations as
accurate as possible, use cookies, or force your visitors to register to
get to the really good stuff. Both of those methods are likely to provide
better estimates, but they are still only estimates.
--
Bill Moran
Potential Technologies
http://www.potentialtech.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is unique? Access stats and IP's
Posted by Leif W <wa...@usa.net>.
Ditto to the points Joshua and Bill made. I've seen the problem of trying
to identify unique visitors from the perspective of a shopping cart
application. The application programmer lost many nights sleep working out
the intricacies of a home-brew solution, which in the end was maybe 95% at
best. You've got to try cookies, server-side credentials (i.e. logins), URL
tokens (but they can be bookmarked or linked URLs), IPs, server-side session
timeouts (all imposed in the application), client-side session timeouts
(cookies, expires, etc.). There were even more intricate checks going on
which I was not privvy to. The case of a shopping cart being able to
uniquely identify a new, or returning visitor is in some regards more
critical than generating stats. The problem that occured sometimes was the
case of incorrect recognition, and revealing of private data to the wrong
party or parties.
Generating a state-layer on top or outside of a fundamentally stateless
protocol is not a trivial matter and is at best only an estimation.
Leif
----- Original Message -----
From: "Eric Frazier" <ef...@kwinternet.com>
To: <us...@httpd.apache.org>
Sent: Thursday, September 04, 2003 8:49 PM
Subject: [users@httpd] What is unique? Access stats and IP's
> Hi,
>
> I just wanted to get an opinon. When you consider the idea of a unique
> vistor, the brain dead way of doing that is to say, one IP one visitor, or
> one cookie one visitor. But aside from cookies, when using IP address, you
> do have to take into account that a vistior from an IP address at 8AM may
> very well not be the same vistor as at 8PM because they are both behind
the
> same firewall/proxy. So one option is to consider that every unique ip
> address within an hour is considered to be a unique vistor. But that seems
> off as well because then if you compare overall stats and compare the
number
> of unique vistors by sorting by unique IP daily, vs the number of unique
> vistors by sorting the IP hourly, the daily number can be like 3 times
more
> than the hourly. So now I am starting to wonder, are we really just left
to
> using cookies for this? What bugs me about that is the stats I see from
our
> sites for Javascript turned off in browsers is actualy pretty high, so if
JS
> is off, I would not at all be shocked to see cookies being rejected as
well.
>
> Maybe the only way is a hybrid of if the cookie was rejected then consider
> the ip, but if there was a cookie for that connection then ignore any IP
data.
>
> If any of the above makes sense to you, and you have an opinon about it I
> would love to hear it.
>
>
> Thanks,
>
> Eric
>
> (250) 655 - 9513 (PST Time Zone)
>
> "Inquiry is fatal to certainty." -- Will Durant
>
>
>
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org