You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Mark Stosberg <ma...@summersault.com> on 2008/10/15 18:31:24 UTC
Re: Any success with storing photos in a database? (prevents
double-submits)
On Tue, 30 Sep 2008 10:06:26 -0400
Mark Stosberg <ma...@summersault.com> wrote:
> On Mon, 29 Sep 2008 17:04:29 -0400
> "Perrin Harkins" <pe...@elem.com> wrote:
>
> > On Mon, Sep 29, 2008 at 3:00 PM, Mark Stosberg <ma...@summersault.com> wrote:
> > > We have a large, busy, database application that relates to millions of
> > > photos, which we also need to store and display.
>
> If I find anything "interesting" about our inconsistency, I'll report back.
I said I'd report back if I found out where the inconsistency was appearing, between files
on the file system and meta-data in the database.
So far, I've found the following:
We had a "double submit" bug that allowed a form to be submitted twice when we
weren't fully prepared for that. We are still researching the best practices to
address this a general case. One approach we are considering is change the
submit action on forms with JavaScript, so it disables the submit button, and
then actually submit the form, preventing one kind of double-submission. It
seems like I don't see this approach happening in the wild much, though. I
suspect there is a better solution.
Beyond this, I think we've found the remaining inconsistencies to be happening
at extremely low rates, to the point where it might be worth completely
tracking down the final issue.
In summary, I think the majority advice found on this list holds true: storing
image files on the file system and meta-data in the database is a good way to
go.
Mark
--
. . . . . . . . . . . . . . . . . . . . . . . . . . .
Mark Stosberg Principal Developer
mark@summersault.com Summersault, LLC
765-939-9301 ext 202 database driven websites
. . . . . http://www.summersault.com/ . . . . . . . .
Re: Any success with storing photos in a database?
Posted by Jonathan Vanasco <jo...@2xlp.com>.
chiming in a few months late...
from my experience, and responding to some thoughts in the thread:
- storing photos in mysql/pgsql is not a good idea. aside from misc
issues that always arise, you end up stressing the db through searches
- the better way would be to store photo meta-data in the system,
and then search for that to serve. its a much leaner search and
faster/more lightweight.
-we played around with a lot of things
-- mogilefs
--- i think flickr also used pre-yahoo. someone mentioned a
flickrbook, there is also a powerpoint from about 1year after they
launched
-- directory hashing
--- create an md5 and store in db; store on filesystem as ab/cd/ef/
gh/abcdefg + suffix ( [original,-thumb ,-medium].jpg )
---- use rewrite rules on nginx to loadbalance across cluster by
bucket and serve
-- i didn't like perlbal, but it can do some neat in-time
bitshifting on gifs to change the color palatte
-- the BEST solution we found and upgraded to was.. Amazon s3 +
Panther Express. this is before amazon offered cdn services
--- store files on amazon -- originals in private bucket, copies in
public buckets
--- serve files from panther, which was a kickass cdn. they cache
everything in their network , which offsets instabilities from amazon
--- don't worry about disk usage, clustering, load balancing,
BACKUP, and miscellaneous stuff designed around images
--- migrating to that setup took us 2 days... and it was a better
performer than 3 months of code and constant maintenance
Re: Any success with storing photos in a database? (prevents double-submits)
Posted by Eric Howe <er...@pieinsky.ca>.
On Oct 15, 2008, at 11:11, Cosimo Streppone wrote:
> On Wed, Oct 15, 2008 at 12:31 PM, Mark wrote:
[...]
> Then Perrin wrote:
>
>> JavaScript is okay, but can be a problem when people hit back
>> expecting to use the form again and the button is still disabled.
>
> You could always trigger some javascript event on page load
> the re-enables the submit control, even if already enabled,
> resetting the flag to "0". This is what I did.
I do it this way (via jQuery) and it works okay even when faced with
the browser's back-button:
function returnFalse() {
return false;
}
$(document).ready(function() {
// Take care of back-button issues first.
$('form').find(':submit').unbind('click', returnFalse);
// Then wire up the auto-disabling stuff.
$('form').submit(function() {
$(this).find(':submit').click(returnFalse);
return true;
});
});
I don't use the 'disabled' attribute as that prevents the browser (at
least some of them) from sending the button back to the server with
the rest of the form.
Eric Howe
eric@pieinsky.ca
Re: Any success with storing photos in a database? (prevents
double-submits)
Posted by Cosimo Streppone <co...@streppone.it>.
On Wed, Oct 15, 2008 at 12:31 PM, Mark wrote:
> We had a "double submit" bug that allowed a form to be submitted twice
> when we weren't fully prepared for that. [...]
> One approach we are considering is change the submit action on forms
> with JavaScript, so it disables the submit button
That's the cheapest option IMO, and it works very well
for 95% of the typical double submit cases (like Windows
users with the "double-click" syndrome :).
On submit event I checked for a flag to be "1".
If "1", then form has been already submitted.
If "0", set to "1" and submit.
Then Perrin wrote:
> JavaScript is okay, but can be a problem when people hit back
> expecting to use the form again and the button is still disabled.
You could always trigger some javascript event on page load
the re-enables the submit control, even if already enabled,
resetting the flag to "0". This is what I did.
Then, if you want to be 100% sure, you have to work
on the server side...
--
Cosimo
Re: Any success with storing photos in a database? (prevents double-submits)
Posted by Michael Peters <mp...@plusthree.com>.
Mark Stosberg wrote:
> So how might an implementation look?
I would either make the uuid the primary key (might affect performance since it's not an integer,
but a string) or a unique key for the same table. Then you don't have anything else to keep track of
(no extra tables, etc).
> Every so often, the table could cleaned up via cron, (since we probably don't
> care about seeing the same UUID weeks apart, just seconds or minutes apart).
UUID's should never collide.
> There is still room for a small race condition in between checking to see if we
> used the UUID and inserting it, but I think that may be acceptable.
If you're really worried about someone attacking you in this way then insert the record with the
uuid first and then let them upload. If you don't find the uuid they are trying to upload to, then
they changed it so just disallow the upload.
--
Michael Peters
Plus Three, LP
Re: Any success with storing photos in a database? (prevents
double-submits)
Posted by Mark Stosberg <ma...@summersault.com>.
On Wed, 15 Oct 2008 13:51:21 -0400
Michael Peters <mp...@plusthree.com> wrote:
> Mark Stosberg wrote:
>
> > At one point in the past I did a variation of this where we put the next ID of
> > a related database sequence in the form, and this would become the new primary
> > key when inserted, and it would of course not allow the same primary key to be
> > used twice. That worked, but I realized was open to abuse if a user tweaked the
> > number to be larger than the sequence. Then, eventually a legitimate user would
> > eventually be assigned that value by the sequence, and it would fail.
> >
> > Are their specific modules that you recommend to help with this?
>
> You can try a UUID (look at Data::UUID). It's not sequential, but someone
> could still tweak the form to create a value that could potentially conflict
> with another value in the future. But I'd say it's much less likely than a
> sequential id.
So how might an implementation look?
We could create table of "form_uuids". Each time a form is submitted, we check
if the UUID we have is has already been used.
If not, we insert the new UUID and proceed.
Every so often, the table could cleaned up via cron, (since we probably don't
care about seeing the same UUID weeks apart, just seconds or minutes apart).
There is still room for a small race condition in between checking to see if we
used the UUID and inserting it, but I think that may be acceptable.
Mark
Re: Any success with storing photos in a database? (prevents double-submits)
Posted by Michael Peters <mp...@plusthree.com>.
Mark Stosberg wrote:
> At one point in the past I did a variation of this where we put the next ID of
> a related database sequence in the form, and this would become the new primary
> key when inserted, and it would of course not allow the same primary key to be
> used twice. That worked, but I realized was open to abuse if a user tweaked the
> number to be larger than the sequence. Then, eventually a legitimate user would
> eventually be assigned that value by the sequence, and it would fail.
>
> Are their specific modules that you recommend to help with this?
You can try a UUID (look at Data::UUID). It's not sequential, but someone could still tweak the form
to create a value that could potentially conflict with another value in the future. But I'd say it's
much less likely than a sequential id.
--
Michael Peters
Plus Three, LP
Re: Any success with storing photos in a database? (prevents
double-submits)
Posted by Mark Stosberg <ma...@summersault.com>.
> Take some elements from the form and search for an add in the past
> <time period>. We did one recently where we took the email address,
> recipient and amount from the form and then made sure that the same
> data hadn't already been submitted in the last 24 hours - working on
> the principle that a user is unlikely to gift the exact same amount to
> the same person on the same day. The check is trivial compared to the
> commit.
>
> FWIW, we looked into this and this approach seemed to be the quickest
> and most reliable available. If you Google on this, you may find an
> article or two which support this approach.
This does seem workable and reliable. I suppose I'm hoping for an
approach that can easy re-used across many forms, without much per-form
effort required.
Also, I think for some kinds of changes being submitted, the check of
"has this happened recently" won't apply.
For example, we allow people to re-order a list of photos. They may well
reverse the ordering a few seconds later and put the photos back in the
original order. (Although that example may not be vurnerable to a
double-submit bug, as it would be valid to submit the same ordering
request twice. )
Mark
Re: Any success with storing photos in a database? (prevents double-submits)
Posted by Simon Forster <si...@ldml.com>.
On 15 Oct 2008, at 18:49, Mark Stosberg wrote:
> On Wed, 15 Oct 2008 12:41:55 -0400
> "Perrin Harkins" <pe...@elem.com> wrote:
>
>> On Wed, Oct 15, 2008 at 12:31 PM, Mark Stosberg
>> <ma...@summersault.com> wrote:
>>> We had a "double submit" bug that allowed a form to be submitted
>>> twice when we
>>> weren't fully prepared for that. We are still researching the best
>>> practices to
>>> address this a general case. One approach we are considering is
>>> change the
>>> submit action on forms with JavaScript, so it disables the submit
>>> button, and
>>> then actually submit the form, preventing one kind of double-
>>> submission. It
>>> seems like I don't see this approach happening in the wild much,
>>> though. I
>>> suspect there is a better solution.
>>
>> JavaScript is okay, but can be a problem when people hit back
>> expecting to use the form again and the button is still disabled.
>
> Thanks for the response.
>
> That's what I was concerned about. I don't have a sense of how
> much this would happen in practice.
Take some elements from the form and search for an add in the past
<time period>. We did one recently where we took the email address,
recipient and amount from the form and then made sure that the same
data hadn't already been submitted in the last 24 hours - working on
the principle that a user is unlikely to gift the exact same amount to
the same person on the same day. The check is trivial compared to the
commit.
FWIW, we looked into this and this approach seemed to be the quickest
and most reliable available. If you Google on this, you may find an
article or two which support this approach.
HTH
Simon
Re: Any success with storing photos in a database? (prevents
double-submits)
Posted by Mark Stosberg <ma...@summersault.com>.
On Wed, 15 Oct 2008 12:41:55 -0400
"Perrin Harkins" <pe...@elem.com> wrote:
> On Wed, Oct 15, 2008 at 12:31 PM, Mark Stosberg <ma...@summersault.com> wrote:
> > We had a "double submit" bug that allowed a form to be submitted twice when we
> > weren't fully prepared for that. We are still researching the best practices to
> > address this a general case. One approach we are considering is change the
> > submit action on forms with JavaScript, so it disables the submit button, and
> > then actually submit the form, preventing one kind of double-submission. It
> > seems like I don't see this approach happening in the wild much, though. I
> > suspect there is a better solution.
>
> JavaScript is okay, but can be a problem when people hit back
> expecting to use the form again and the button is still disabled.
Thanks for the response.
That's what I was concerned about. I don't have a sense of how
much this would happen in practice.
> Another approach is a unique ID in the form that you track in the
> user's session (i.e. this ID was seen before). If the problem is
> large uploads with no feedback until they finish, you can use one of
> the upload progress tools.
At one point in the past I did a variation of this where we put the next ID of
a related database sequence in the form, and this would become the new primary
key when inserted, and it would of course not allow the same primary key to be
used twice. That worked, but I realized was open to abuse if a user tweaked the
number to be larger than the sequence. Then, eventually a legitimate user would
eventually be assigned that value by the sequence, and it would fail.
Are their specific modules that you recommend to help with this?
Mark
Re: Any success with storing photos in a database? (prevents double-submits)
Posted by Perrin Harkins <pe...@elem.com>.
On Wed, Oct 15, 2008 at 12:31 PM, Mark Stosberg <ma...@summersault.com> wrote:
> We had a "double submit" bug that allowed a form to be submitted twice when we
> weren't fully prepared for that. We are still researching the best practices to
> address this a general case. One approach we are considering is change the
> submit action on forms with JavaScript, so it disables the submit button, and
> then actually submit the form, preventing one kind of double-submission. It
> seems like I don't see this approach happening in the wild much, though. I
> suspect there is a better solution.
JavaScript is okay, but can be a problem when people hit back
expecting to use the form again and the button is still disabled.
Another approach is a unique ID in the form that you track in the
user's session (i.e. this ID was seen before). If the problem is
large uploads with no feedback until they finish, you can use one of
the upload progress tools.
- Perrin