Erlang Filesystem errors and Couchdb Performance

[david martin <da...@lymegreen.co.uk>]

I quote from <er...@erlang.org>
Wed Oct 14 20:50:05 CEST 2009

"Erlang on EC2 - Filesystem errors? from  Paul Davis
Hey list, I've got a bit of a head scratcher.
The basic premise is trying to run
Erlang on an EC2 sometimes results in errors like those pasted below.
I've seen this type of error from two different CouchDB users and the
only thing I can figure is that they're both using EC2.
Granted lots of other people use Erlang on EC2 just fine so I haven't the slightest
if that's even related.
The only thing that's confusing me is how the VM even started if it's
unable to read these files. My first thought was that some rogue
backup process was changing file permissions as it did its thing but
both users reported nothing in cron or had knowledge of anything of
that nature running on their system.
I've googled around looking for general filesystem issues on EC2 as
well as things related specifically to Erlang. The only thing that
looked remotely similar was very similar output that some rabbitmq
users have reported. The most interesting of which is this thread [1]
where the last post says that the guy could fix the errors just by
running a 'find | grep sth' which seems odd.
Anyone else experienced this before? Thank you,
Paul J. Davis"

"File operation error: eacces. Target: ..
File operation error: eacces. Target: ./user_sup.beam.
File operation error: eacces. Target: ./supervisor_bridge.beam.
File operation error: eacces. Target: ./user.beam.
File operation error: eacces. Target: ./kernel_config.beam.
File operation error: eacces. Target: ./queue.beam.
File operation error: eacces. Target: .. Function: read_file_info.
File operation error: eacces. Target: ./user_sup.beam.
File operation error: eacces. Target: ./supervisor_bridge.beam.
File operation error: eacces. Target: ./user.beam.
File operation error: eacces. Target: ./kernel_config.beam.
File operation error: eacces. Target: ./queue.beam.
File operation error: eacces. Target: ./error_logger_tty_h.beam.
File operation error: eacces. Target: ./io_lib.beam.
File operation error: eacces. Target: ./io_lib_format.beam.
File operation error: eacces. Target: ./io_lib_pretty.beam.
File operation error: eacces. Target: ./io.beam.
File operation error: eacces. Target: ./c.beam.
File operation error: eacces. Target: ./file.beam.
File operation error: eacces. Target: ./erl_eval.beam.
File operation error: eacces. Target: ./orddict.beam.
File operation error: eacces. Target: ./file_io_server.beam.
File operation error: eacces. Target: ./erl_posix_msg.beam.file:path_eval([".","/home/couchdb"],".erlang"): permission denied
File operation error: eacces. Target: ./couch.beam.
File operation error: eacces. Target: ..
File operation error: eacces. Target: ./erl_scan.beam.
File operation error: eacces. Target: ./erl_parse.beam."

For me the most interesting aspect of this posting was

"The only thing that's confusing me is how the VM even started if it's
unable to read these files."

and that the thread was unresolved.

I am running a KVM virtual instance of Arch Linux (the VM) in an Arch Linux Host (the Host).
The VM is a basic Arch Linux distribution running the latest Erlang R14B and Couchdb 1.1.0.
The Couchdb has all database, view and log files mounted as NFS on a separate freeNAS (bsd) server.
The image of the VM is served from the host.

As you can imagine the ownership and permissions
of the various parts has been a headache which I have not completely worked out.
In my efforts to get this set-up to work, I found the above post, which seemed similar,
but it has nothing to do with EC2, as the original post suspected.

Couchdb on VM has normally files with  permissions Couchdb:Daemon
I have transferred /var/lib/couchdb to NFS share with  permissions Couchdb:Daemon
the VM is served from an image from a folder root:root from a file root:root.

When Couchdb is started in the VM with the deamon, It starts and operates normally.
That is a Couchapp on the database on the NFS share is accessed works to serve a webpage,
and updates the database in response to actions from the page javascript.

EXCEPT on viewing the logs the  "File operation error: eacces" errors occur as listed below.

If on the same setup Couchdb is started as root from the console everything works as before
but there are no "File operation error: eacces" errors.

Does this imply that Couchdb is trying to access files that it does not need and has already?.

If this is the case then it must have a performance penalty for all actions taken by Couchdb to
access files it does not need.

  File operation error: eacces. Target: ./couch_rep_sup.beam.
  File operation error: eacces. Target: ./httpd_util.beam.
  File operation error: eacces. Target: ./couch_task_status.beam.
  File operation error: eacces. Target: ./couch_file.beam.
  File operation error: eacces. Target: ./filelib.beam.
  File operation error: eacces. Target: ./couch_view.beam.
  File operation error: eacces. Target: ./couch_db_update_notifier.beam.
  File operation error: eacces. Target: ./couch_uuids.beam.
  File operation error: eacces. Target: ./crypto.beam.
  File operation error: eacces. Target: ./couch_db_update_notifier_sup.beam.
  File operation error: eacces. Target: ./couch_os_daemons.beam.
  File operation error: eacces. Target: ./couch_stats_collector.beam.
  File operation error: eacces. Target: ./couch_query_servers.beam.
  File operation error: eacces. Target: ./couch_stats_aggregator.beam.
  File operation error: eacces. Target: ./couch_replication_manager.beam.
  File operation error: eacces. Target: ./couch_rep.beam.
  File operation error: eacces. Target: ./couch_db.beam.
  File operation error: eacces. Target: ./couch_db_updater.beam.
  File operation error: eacces. Target: ./couch_btree.beam.
  File operation error: eacces. Target: ./couch_ref_counter.beam.
  File operation error: eacces. Target: ./couch_key_tree.beam.
  File operation error: eacces. Target: ./couch_doc.beam.
  File operation error: eacces. Target: ./couch_changes.beam.
  File operation error: eacces. Target: ./couch_httpd.beam.
  File operation error: eacces. Target: ./couch_httpd_view.beam.
  File operation error: eacces. Target: ./eval_bits.beam.
  File operation error: eacces. Target: ./mochiweb_http.beam.
  File operation error: eacces. Target: ./mochilists.beam.
  File operation error: eacces. Target: ./mochiweb_socket_server.beam.
  File operation error: eacces. Target: ./sets.beam.
  File operation error: eacces. Target: ./mochiweb_socket.beam.
  File operation error: eacces. Target: ./gen_tcp.beam.
  File operation error: eacces. Target: ./inet_tcp.beam.
  File operation error: eacces. Target: ./mochiweb_acceptor.beam.
  File operation error: eacces. Target: ./couch_auth_cache.beam.
  File operation error: eacces. Target: ./couch_httpd_vhost.beam.
  File operation error: eacces. Target: ./couch_external_manager.beam.
  [info] [<0.31.0>] Apache CouchDB has started on http://0.0.0.0:5984/
File operation error: eacces. Target: ./mochiweb.beam.
File operation error: eacces. Target: ./mochiweb_headers.beam.
File operation error: eacces. Target: ./mochiweb_request.beam.
File operation error: eacces. Target: ./mochiweb_util.beam.
[debug] [<0.872.0>] 'GET' /audio/_design/audio/mixer1.html {1,1} from "192.168.1.10"
File operation error: eacces. Target: ./couch_httpd_oauth.beam.
File operation error: eacces. Target: ./oauth_uri.beam.
[debug] [<0.872.0>] OAuth Params: []
File operation error: eacces. Target: ./couch_httpd_auth.beam.
File operation error: eacces. Target: ./mochiweb_cookies.beam.
File operation error: eacces. Target: ./base64.beam.
[debug] [<0.872.0>] timeout 43200
File operation error: eacces. Target: ./couch_httpd_db.beam.
[info] [<0.872.0>] 192.168.1.10 - - 'GET' /audio/_design/audio/mixer1.html 200
File operation error: eacces. Target: ./mochiweb_response.beam.
File operation error: eacces. Target: ./couch_stream.beam.
[debug] [<0.872.0>] 'GET' /audio/_design/audio/style/mixer.css {1,1} from "192.168.1.10"
etc...

In spite of all these errors the Couchapp works perfectly!
Must be the magic of Erlang!


David M. W. Martin

Re: Erlang Filesystem errors and Couchdb Performance

[Randall Leeds <ra...@gmail.com>]

You might both check that the CouchDB user has access to its home directory.
I think if it doesn't the default init script might fail, but if
you're launching CouchDB yourself using your own daemon monitoring
system and it doesn't have access to its home directory (but
permissions are right for the log, lib, etc dirs), it might be that
Erlang is checking the current directory first (hence the ./) when
trying to load modules and failing. It would then find those modules
later on down the module path, and so it doesn't cause a problem.

I suggest looking at the CouchDB user's home directory assuming that's
the current working directory the way you're launching it, but it may
be a different directory that's current, depending on you run things.

On Wed, Jun 29, 2011 at 09:03, Paul Davis <pa...@gmail.com> wrote:
>> For me the most interesting aspect of this posting was
>>
>> "The only thing that's confusing me is how the VM even started if it's
>> unable to read these files."
>>
>> and that the thread was unresolved.
>
> David,
>
> As it turns out the cause of the issue I was reporting was totally
> unrelated and ended up being caused by a Chef script copying the
> ._couch_module.beam files that OS X produces with the fancy extended
> metadata.
>
> As far as I can tell on these eaccess issues they're mostly just
> noise. I never got an explanation (or one that I am able to remember)
> on where these errors come from and so I mostly just ignore them.
>
> HTH,
> Paul Davis
>

Re: Erlang Filesystem errors and Couchdb Performance

[Paul Davis <pa...@gmail.com>]

> For me the most interesting aspect of this posting was
>
> "The only thing that's confusing me is how the VM even started if it's
> unable to read these files."
>
> and that the thread was unresolved.

David,

As it turns out the cause of the issue I was reporting was totally
unrelated and ended up being caused by a Chef script copying the
._couch_module.beam files that OS X produces with the fancy extended
metadata.

As far as I can tell on these eaccess issues they're mostly just
noise. I never got an explanation (or one that I am able to remember)
on where these errors come from and so I mostly just ignore them.

HTH,
Paul Davis