You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Paul Davis <pa...@gmail.com> on 2008/11/23 08:44:51 UTC
Closing Files
Hey,
I've been hacking on jchris's update-false branch for the past couple
days and I've finally gotten everything to the point of passing all
the Futon tests. But I've run into a bit of an issue. The tests all
pass the first time through, but when the tests are repeated before a
server restart, I start getting EMFILE errors indicating i've
exhausted file handles. I cannot for the life of me piece together how
file handles are cleaned up *after* the view has been completely
written to the client.
Hopefully I'm not just tired and missing something obvious, but can
anyone give me a description of the expected open/close states for the
file descriptors used for both the couch_db and the #group.fd handles?
Specifically when they change and how that gets triggered.
Thanks,
Paul
Re: Closing Files
Posted by Paul Davis <pa...@gmail.com>.
On Sun, Nov 23, 2008 at 3:05 PM, Chris Anderson <jc...@apache.org> wrote:
> On Sun, Nov 23, 2008 at 11:54 AM, Paul Davis
> <pa...@gmail.com> wrote:
>> Doing a couple tests it looks like mochiweb is using a single process
>> for an entire Keep-Alive connection. Thus when under load from the
>> Futon tests, it can still run out of descriptors. Testing from curl
>> has the expected effect of closing the FD for every request.
>
> This sounds like our Couch. ;)
>
>>
>> The next two ideas I had were to use either an LRU cache similar to
>> couch_server.erl or add a try/after block around requests in
>> couch_httpd_view.erl. I'm starting to lean towards the slight changes
>> to couch_httpd_view.erl at the moment because it seems to be the
>> better of the two even though I was trying to keep changes to it as
>> minimal as possible. (I've changed about 5-10 lines so far, and those
>> were mostly changing module names on function calls.)
>>
>> Anyone got any input?
>>
>
> Keeping the changes near the view code sounds smart. I've pulled your
> branch, but I'm driving up to Seattle for the night so I don't know
> when I'll have a chance to investigate.
>
Yeah. After looking at the http view code I think I can add the
appropriate logic in about 8 lines so I think that's what I'm gonna
run with. Hopefully I'll have all of the view logic banged out in an
hour or two.
> Chris
>
>
> --
> Chris Anderson
> http://jchris.mfdz.com
>
Re: Closing Files
Posted by Chris Anderson <jc...@apache.org>.
On Sun, Nov 23, 2008 at 11:54 AM, Paul Davis
<pa...@gmail.com> wrote:
> Doing a couple tests it looks like mochiweb is using a single process
> for an entire Keep-Alive connection. Thus when under load from the
> Futon tests, it can still run out of descriptors. Testing from curl
> has the expected effect of closing the FD for every request.
This sounds like our Couch. ;)
>
> The next two ideas I had were to use either an LRU cache similar to
> couch_server.erl or add a try/after block around requests in
> couch_httpd_view.erl. I'm starting to lean towards the slight changes
> to couch_httpd_view.erl at the moment because it seems to be the
> better of the two even though I was trying to keep changes to it as
> minimal as possible. (I've changed about 5-10 lines so far, and those
> were mostly changing module names on function calls.)
>
> Anyone got any input?
>
Keeping the changes near the view code sounds smart. I've pulled your
branch, but I'm driving up to Seattle for the night so I don't know
when I'll have a chance to investigate.
Chris
--
Chris Anderson
http://jchris.mfdz.com
Re: Closing Files
Posted by Paul Davis <pa...@gmail.com>.
After waking up and poking around trunk with a few log statements I
managed to unravel a clearer picture of what's going on.
Near as I can tell the update_loops have a 10 second timeout in
get_notify_pids and when this fails it cascades and ends up closing
the file handles.
Instead of first trying to directly mirror that paradigm i attempted
to use the reference counting semantics available in couch_file to
have files automatically closed when a request ends by upping the ref
count when a process registers to use an update group. This almost
works.
Doing a couple tests it looks like mochiweb is using a single process
for an entire Keep-Alive connection. Thus when under load from the
Futon tests, it can still run out of descriptors. Testing from curl
has the expected effect of closing the FD for every request.
The next two ideas I had were to use either an LRU cache similar to
couch_server.erl or add a try/after block around requests in
couch_httpd_view.erl. I'm starting to lean towards the slight changes
to couch_httpd_view.erl at the moment because it seems to be the
better of the two even though I was trying to keep changes to it as
minimal as possible. (I've changed about 5-10 lines so far, and those
were mostly changing module names on function calls.)
Anyone got any input?
On Sun, Nov 23, 2008 at 3:12 AM, Paul Davis <pa...@gmail.com> wrote:
> Totally forgot to paste the link after I explicitly pushed code.
>
> http://github.com/davisp/couchdb/tree/update-false
>
> Purge works fine for me.
>
> Some things to note about the code:
>
> I finally made a couch_os_process that can be used for the cases where
> we have external processes like the action2 external stuff.
>
> The overall restructuring looks like this:
>
> 1. couch_view sets up couch_view_group gen_servers that perform the
> actual update.
> 2. couch_view_group 'owns' a single view_updater process that
> basically iterates over enum_docs_since_seq and throws them at
> couch_query_server
> 3. couch_query_server cast's incoming docs to N couch_os_processes
> round robin style.
> 4. Using a closure, the couch_os_process's send the key/value output
> back to couch_view_group which buffers and flushes to disk.
> 5. When couch_view_group recieves as many results as view_updater sent
> out, it terminates sending the final group to any process that
> registered to get a copy of it.
>
> Also, at the moment I think this implementation might be slower than
> the original, but I have a sneaking suspicion that I'm calling fsync
> quite a few times that I shouldn't be.
>
> update-false isn't actually in yet, but it'll fit nicely into
> couch_view:get_map_view with a call to get the current version of the
> Group record stored in the gen_server state.
>
> Also, I realized it would be fairly trivial to add a wait=N to wait at
> most N seconds in case people think that's a good idea. I'm on the
> fence.
>
> And it's time for bed before my brain explodes.
>
> On Sun, Nov 23, 2008 at 2:51 AM, Chris Anderson <jc...@apache.org> wrote:
>> On Sat, Nov 22, 2008 at 11:44 PM, Paul Davis
>> <pa...@gmail.com> wrote:
>>> I start getting EMFILE errors indicating i've
>>> exhausted file handles. I cannot for the life of me piece together how
>>> file handles are cleaned up *after* the view has been completely
>>> written to the client.
>>>
>>
>> I had the same error, but it would show up when attempting to view
>> after a purge (which would never return to the client), so I bet it
>> was a different cause. (Mine kept spawning new view updates, over and
>> over, due to an off-by-one error when the seq-id decremented)
>>
>> Is your version of code up on a github branch?
>>
>> Chris
>>
>>
>> --
>> Chris Anderson
>> http://jchris.mfdz.com
>>
>
Re: Closing Files
Posted by Paul Davis <pa...@gmail.com>.
Totally forgot to paste the link after I explicitly pushed code.
http://github.com/davisp/couchdb/tree/update-false
Purge works fine for me.
Some things to note about the code:
I finally made a couch_os_process that can be used for the cases where
we have external processes like the action2 external stuff.
The overall restructuring looks like this:
1. couch_view sets up couch_view_group gen_servers that perform the
actual update.
2. couch_view_group 'owns' a single view_updater process that
basically iterates over enum_docs_since_seq and throws them at
couch_query_server
3. couch_query_server cast's incoming docs to N couch_os_processes
round robin style.
4. Using a closure, the couch_os_process's send the key/value output
back to couch_view_group which buffers and flushes to disk.
5. When couch_view_group recieves as many results as view_updater sent
out, it terminates sending the final group to any process that
registered to get a copy of it.
Also, at the moment I think this implementation might be slower than
the original, but I have a sneaking suspicion that I'm calling fsync
quite a few times that I shouldn't be.
update-false isn't actually in yet, but it'll fit nicely into
couch_view:get_map_view with a call to get the current version of the
Group record stored in the gen_server state.
Also, I realized it would be fairly trivial to add a wait=N to wait at
most N seconds in case people think that's a good idea. I'm on the
fence.
And it's time for bed before my brain explodes.
On Sun, Nov 23, 2008 at 2:51 AM, Chris Anderson <jc...@apache.org> wrote:
> On Sat, Nov 22, 2008 at 11:44 PM, Paul Davis
> <pa...@gmail.com> wrote:
>> I start getting EMFILE errors indicating i've
>> exhausted file handles. I cannot for the life of me piece together how
>> file handles are cleaned up *after* the view has been completely
>> written to the client.
>>
>
> I had the same error, but it would show up when attempting to view
> after a purge (which would never return to the client), so I bet it
> was a different cause. (Mine kept spawning new view updates, over and
> over, due to an off-by-one error when the seq-id decremented)
>
> Is your version of code up on a github branch?
>
> Chris
>
>
> --
> Chris Anderson
> http://jchris.mfdz.com
>
Re: Closing Files
Posted by Chris Anderson <jc...@apache.org>.
On Sat, Nov 22, 2008 at 11:44 PM, Paul Davis
<pa...@gmail.com> wrote:
> I start getting EMFILE errors indicating i've
> exhausted file handles. I cannot for the life of me piece together how
> file handles are cleaned up *after* the view has been completely
> written to the client.
>
I had the same error, but it would show up when attempting to view
after a purge (which would never return to the client), so I bet it
was a different cause. (Mine kept spawning new view updates, over and
over, due to an off-by-one error when the seq-id decremented)
Is your version of code up on a github branch?
Chris
--
Chris Anderson
http://jchris.mfdz.com