You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by "Dondi M. Stroma" <ds...@verizon.net> on 2006/10/03 03:18:24 UTC
mystery caching problem
Hi,
Before I begin, I want to let you know that I've already poured over all
mod_perl/Apache/CGI/perl related docs, porting guides, mailing list
archives, etc. I've read and re-read about the traps and pitfalls of
mod_perl. I have "use strict" on, warnings on, and I've quadrouple checked
for closures, global variables, "variable will not stay shared" warnings,
always passing variables to subroutines as parameters, etc.
Well, I've been experiencing a mysterious problem for a very long time now
with a web app that I wrote. It was written to work under either
mod_perl+Apache::Registry or mod_cgi, but obviously I'm running it under
mod_perl or I wouldn't be posting this message here. It's running on Fedora
with Apache 1.33 and mod_perl 1.29 (dso). The scripts use CGI.pm (latest
version) for sending headers, and getting and retrieving cookies which store
username and encrypted password. I'm not using any kind of reverse proxy or
caching modules or anything like that, although I am using mod_rewrite for
clean URLs (I know I could use a handler for that, but like I said
everything was written to work under both mod_perl and regular mod_cgi).
The problem: occasionally, when a user logs in, they end up logging in to
someone else's account! But it happens very, very infrequently, and I've
been unable to even reproduce the problem myself. The application averages
about 5000 logins per day, and users report this problem only about once
every two weeks. So, this problem happens only 1 in 70,000 times. I would
think that if there was a bug in my scripts, it would be happening much more
than 0.0014% of the time.
Here's where it get's interesting. CGI.pm has an undocumented method called
cache() which, if called, outputs the "Pragma: no-cache" header. I found it
confusing, so just modified my copy of CGI.pm to always print Pragma:
no-cache whenever the header() method is called. It seemed to fix the
problem! I didn't get any "wrong account" reports for several months. A
few days ago, the problem came back; but then I realized a recent automatic
update to CGI.pm had overridden my hack... which seems to confirm that the
"Pragma: no-cache" header made the problem go away.
But rather than speculating, I'd like to know what is *really* going on
here. I still can't figure out why the problem was occurring in the first
place, or why the no-cache header *seems* to fix it. Does anyone have any
ideas? I'd like to understand and fix the problem for good rather than
slapping band-aids on it.
Re: mystery caching problem
Posted by David Nicol <da...@gmail.com>.
> find out if these people are getting into the system because they're
> sending a cookie/login string with the right details for the wrong
> account
using one-time-use capability keys that can only be obtained by reading
the user's e-mail will prevent erroneous log-ins while reducing the
number of credentials the users need to remember.
Janes Doe and Smith may not be able to say who is who in a mirror
but its not likely that they actually receive each other's e-mail.
bitcard and AIS
do essentially the same thing. I will be totally thrilled to offer a mod_perl
enabled AIS::client if anyone writes one; I don't know if Authen::Bitcard
works right under mod_perl but I expect it does. Ask is much more reliable
than I am.
--
The Country Of The Blind, by H.G. Wells
http://cronos.advenge.com/pc/Wells/p528.html
Re: mystery caching problem
Posted by Jonathan Vanasco <mo...@2xlp.com>.
On Oct 3, 2006, at 4:01 PM, Dondi M. Stroma wrote:
> One time, two different users reported this problem at roughly the
> same time. But what is interesting is that not only did they both
> enter *a* wrong account, they both entered the *same* wrong account!
are you sure the login routine is working?
the db select could have an issue with it
the password matching could have an issue with it
did you look into how that could have legitimately happened? maybe
these people are making typos and entering the wrong accounts, but in
a right way. i' hear tons of issues stemming from eager auto-
complete fields in safari and firefox.
as i mentioned before on this list, an agency i once worked at made
the brilliant move of taking a contract to handle a 1.800 line for
salon beauty products in addition to the online account management
help line we had to support.
the one lesson i learned from occasionally getting a 'general 1800'
question sent to me, is that people are stupid. really stupid.
really, really , really really really stupid.
on an average day, I would get 10 each of these 2 question/answers
1- Q. I lost my password. I think my account is hairdresser1. My
email is hairdresser1@hotmail.com. i always use the password
######. its not working.
Solution: I altered the db to also contain a md5(username) and md5
(email) field. guess how many of them matched?
literally, 50% of the problems were because the person used their
email or username as their password. and then forgot. the other
50%....
2- Q. I logged in, and I'm in someone elses account. This is
broken. My privacy is broken.
Solution: Looking at the new db fields above, what do we see?
the bulk of people, for a hairstyling site, had usernames relating
to hair, with hair in them.
Lets say the person who called was Jane Doe from New York
The database would have:
username password
=========|=========
hairjane | jane
janecuts | jane
janecuts | scissors
janestyle | scissors
nycjane | shampoo
which user is she? which password? why is everything so similar?
i think about 80% of the passwords that weren't tied to the name /
address of the user came from a pool of 20 salon related words.
Now lets say that Jane Smith is registered. And she's also from
new york.
guess what.. jane smith AND jane doe - more often than you would
expect - would think that they were each other. they'd forget their
name, remember it must have been hair-something/name-something, and
come up with a name that is registered to someone else. then they'd
take a stab at the password, and usually get it right by the 4th or
5th time.
so i don't know... if your logins are caused by user email- or
something that is hard to 'accidentally' come by- then you may very
well be having a technical problem.
but if your system is using a username, its a very real possibility
that your client base isn't that 'security minded', and making
random lucky guesses at passwords and user names that actually do work.
( for what its worth.... we tried using 'strong' passwords, assigned
passwords, and varying schemes to ensure that people couldn't log in
as each other. whenever we did, people stopped using the system.
they hated dealing with passwords they couldn't remember, couldn't
remember where they put the passwords they wrote down, and were too
annoyed with the reset feature. so we rolled back to insecure
passwords ( but migrated out any sensitve info ), and usage went
right back up. people who came across one of these "security bugs"
would actually be nice... and when we feigned 'oh no not another bug'
they would be very apologetic )
> And we have almost 100,000 users in our database! The script
> keeps a log of the numerical user ID, IP address, and timestamp of
> each successful user to login, and whenever someone reports
> entering the wrong account, nothing unusual or having a pattern
> ever shows up there either. If something was wrong with the login
> logic, I would expect to see an entry with the user ID of one user
> and the IP of the other user. That's never happened.
i'd suggest logging the form paramters-- get a record of username +
md5(password)
i think you need to track down where/how this problem is manifesting
itself. lets say it is because of some weird caching issue done by
an ISP-- you're not going to be able to fix that.
but on your end, you might be able to do some things to code around it.
i'd keep a log of account attempts for the next few weeks. log what
you've been logging already... but also log in the cookie values, the
form values, and the return values.
find out if these people are getting into the system because they're
sending a cookie/login string with the right details for the wrong
account , if something- somewhere- in your code is ignoring what they
send and using old values, or if something in your code is rewriting
their info with other values.
personally, i'd log
1. incoming form values ( cookie /login page )
2. form values on your server that you submit for verification
3. the verification results
4. the values ( cookie / whatever ) you send back
i'd also log an md5 of every successful username and password combo ,
and go though that looking for errors-- and similarites. you might
just have some popular usernames / passwords.
also, 'normal' isps like comcast etc do a lot of caching and
proxying. especially for dialup users. never eliminate them as
being a source of your problems. especially if their name rhymes
with verizon.
// Jonathan Vanasco
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| FindMeOn.com - The cure for Multiple Web Personality Disorder
| Web Identity Management and 3D Social Networking
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| RoadSound.com - Tools For Bands, Stuff For Fans
| Collaborative Online Management And Syndication Tools
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
Re: mystery caching problem
Posted by "Dondi M. Stroma" <ds...@verizon.net>.
Jonathan Vanasco wrote:
> On Oct 3, 2006, at 1:30 AM, David Emery wrote:
>> Just a shot in the dark, but could it be a problem with your log-in
>> process
>> rather than a caching issue? Like maybe the unique value you're basing
>> the
>> user's identity on isn't quite as unique as you think?
>
> I'm going to agree with this being AT LEAST the first place to shift
> your focus.
>
> Since the issue you have happens so rarely, you don't really know if the
> pragma thing has anything to do within it. All of your tests seem to be
> within a margin of error.
>
> How exactly are you logging people in ? How are they identified in a
> database? How are they identified to the server after they have been
> verified through the db? How are cookies being made / handled / sent to
> the server ?
>
> Is it possible that you're experiencing improbable but fully possible
> collisions in md5 values?
>
> Also, have you been debugging the actual login logic, and then going
> through logs to match up what was happening with an error? there could
> be some odd issue where people have similar passwords or logins, and the
> verification code sucks.
I have thoroughly checked the login logic, and the evidence really does
seems to point to some kind of caching. Like you said, I can't be
completely sure, so I'll describe how the login process works. Users enter
their username and password into a login form which POSTs to a Registry
script. The script gets the parameters via CGI.pm, runs a few regexes to
check for strange characters, and calls a sub named validate_user(), passing
the username and password to it. The sub executes a database query to fetch
a row matching the username, if any, and makes sure the fetched user/pass
match the form user/pass. A hash is returned with all of the user's
account info (if they don't match, the sub returns 0, and the script prints
a typical wrong password message).
I've done many diagnosics on this. For example, I added some code to the
script that, right before sending the header, decrypts the cookie to make
sure that it matches the form params.
One time, two different users reported this problem at roughly the same
time. But what is interesting is that not only did they both enter *a*
wrong account, they both entered the *same* wrong account! And we have
almost 100,000 users in our database! The script keeps a log of the
numerical user ID, IP address, and timestamp of each successful user to
login, and whenever someone reports entering the wrong account, nothing
unusual or having a pattern ever shows up there either. If something was
wrong with the login logic, I would expect to see an entry with the user ID
of one user and the IP of the other user. That's never happened.
I'm using two way encryption (using the Crypt::CBC module) instead of MD5,
so collisions aren't an issue. The username and password are both encrypted
individually. The set-cookie header is generated like this:
my $cookieobject = $cgi->cookie(-name => 'logininfo', -value =>
[($encrypted_user, $encrypted_pass)]);
print
$cgi->redirect(-cookie=>$cookieobject, -location=>'/youarenowloggedin');
On visits to other pages, the user and pass are retrieved from the cookie,
decrypted, and revalidated using the same validate_user() sub as before.
Now that I think of it, I wonder if issuing a cookie in a redirect header is
causing a problem? I checked the response headers, and it returns a 302
status code. Maybe I should change it to 303? Even if that is a problem, I
don't understand where or how the alleged caching is taking place.
Re: mystery caching problem
Posted by Jonathan Vanasco <mo...@2xlp.com>.
On Oct 3, 2006, at 1:30 AM, David Emery wrote:
> Just a shot in the dark, but could it be a problem with your log-in
> process
> rather than a caching issue? Like maybe the unique value you're
> basing the
> user's identity on isn't quite as unique as you think?
I'm going to agree with this being AT LEAST the first place to shift
your focus.
Since the issue you have happens so rarely, you don't really know if
the pragma thing has anything to do within it. All of your tests
seem to be within a margin of error.
How exactly are you logging people in ? How are they identified in
a database? How are they identified to the server after they have
been verified through the db? How are cookies being made / handled /
sent to the server ?
Is it possible that you're experiencing improbable but fully possible
collisions in md5 values?
Also, have you been debugging the actual login logic, and then going
through logs to match up what was happening with an error? there
could be some odd issue where people have similar passwords or
logins, and the verification code sucks.
Re: mystery caching problem
Posted by David Emery <da...@skiddlydee.com>.
On Date: Mon, 02 Oct 2006 21:18:24 -0400, "Dondi M. Stroma" <ds...@verizon.net> wrote:
> The problem: occasionally, when a user logs in, they end up logging in to
> someone else's account! But it happens very, very infrequently, and I've
> been unable to even reproduce the problem myself. The application averages
> about 5000 logins per day, and users report this problem only about once
> every two weeks. So, this problem happens only 1 in 70,000 times. I would
> think that if there was a bug in my scripts, it would be happening much more
> than 0.0014% of the time.
Just a shot in the dark, but could it be a problem with your log-in process
rather than a caching issue? Like maybe the unique value you're basing the
user's identity on isn't quite as unique as you think?
Dave
Re: mystery caching problem
Posted by "Dondi M. Stroma" <ds...@verizon.net>.
Perrin Harkins wrote:
>
> Is there a proxy server in front of your mod_perl? You have to be careful
> to avoid caching of Set-Cookie headers in the proxy.
>
> - Perrin
Nope...I wish that was the case because it would explain a lot.
Occasionally people use anonymous proxies to access our site, but the IP
addresses of users who have reported this problem, as well as the IPs of the
"wrong user" were all from normal ISPs like Comcast, etc.
One thing I didn't mention is that we're running off a virtual private
server (VPS using Virtuozzo), but it's supposed to be just like a dedicated
server. I don't know how that could cause it.
Re: mystery caching problem
Posted by Perrin Harkins <pe...@elem.com>.
Dondi M. Stroma wrote:
> The problem: occasionally, when a user logs in, they end up logging in
> to someone else's account!
[...]
> Here's where it get's interesting. CGI.pm has an undocumented method
> called cache() which, if called, outputs the "Pragma: no-cache" header.
> I found it confusing, so just modified my copy of CGI.pm to always print
> Pragma: no-cache whenever the header() method is called. It seemed to
> fix the problem!
Is there a proxy server in front of your mod_perl? You have to be
careful to avoid caching of Set-Cookie headers in the proxy.
- Perrin