[jira] [Comment Edited] (HAWQ-1210) Documentation regarding usage of libhdfs3 in concurrent environment

["William Forson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HAWQ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15738797#comment-15738797 ] 

William Forson edited comment on HAWQ-1210 at 12/11/16 1:12 AM:
----------------------------------------------------------------

Since my own use case involves lots of RAII wrappers and extra context, I've created a standalone executable that uses libhdfs3 to repeatedly download 2 files in separate threads.

My intent is to use instances of this test program as input to dynamic analysis tools such as valgrind DRD. This is actually my first time using valgrind's DRD tool, but the output seems suggestive -- namely, even when I run my test program with {{-i 1}} (i.e. downloading each file only once), the DRD output is indicating anywhere from 35 to 60 errors (for both {{-shared}} and {{-separate}}, though {{-separate}} generally seems to produce a somewhat larger number of errors).

Of course, the operative word there is "suggestive". As this is c/c++, who knows how many platform-specific variables might be coming into play here. Also, perhaps people who work on this project already have similar/better test programs. At any rate, I'm sharing this just in case it might be useful to anyone on your side.


was (Author: wdf):
Since my own use case involves lots of RAII wrappers and extra context, I've created a standalone executable that uses libhdfs3 to repeatedly download 2 files in separate threads.

My intent is to use instances of this test program as input to dynamic analysis tools such as valgrind DRD. This is actually my first time using valgrind's DRD tool, but the output seems suggestive -- namely, even when I run my test program with `-i 1` (i.e. downloading each file only once), the DRD output is indicating anywhere from 35 to 60 errors (for both `-shared` and `-separate`, though `-separate` generally seems to produce a somewhat larger number of errors).

Of course, the operative word there is "suggestive". As this is c/c++, who knows how many platform-specific variables might be coming into play here. Also, perhaps people who work on this project already have similar/better test programs. At any rate, I'm sharing this just in case it might be useful to anyone on your side.

> Documentation regarding usage of libhdfs3 in concurrent environment
> -------------------------------------------------------------------
>
>                 Key: HAWQ-1210
>                 URL: https://issues.apache.org/jira/browse/HAWQ-1210
>             Project: Apache HAWQ
>          Issue Type: Bug
>          Components: libhdfs
>            Reporter: William Forson
>            Assignee: Lei Chang
>         Attachments: hdfs_fs_concurrent_test.cpp
>
>
> Hi,
> I've been using libhdfs3 in a single-threaded environment for several months now, without any problems. However, as soon as I tried using the library concurrently from multiple threads: hello, segfaults.
> Although the source of these segfaults is annoyingly subtle, I've managed to isolate it to a relatively small block of my code that does nothing interesting aside from using libhdfs3 to download a single hdfs file.
> To be clear: I assume that the mistake here is mine -- that is, that I am using your library incorrectly. However, I have been unable to find any documentation as to how the libhdfs3 API _should_ be used in a multi-threaded environment. I initially interpreted this to mean, "go to town, it's all more or less thread-safe", but I am now questioning that interpretation.
> So, I have a question, and a request.
> Question: Are there any known, non-obvious concurrency gotchas regarding the usage of libhdfs3 (or whatever it's currently called)?
> Request: Could you please add some documentation, to the README and/or hdfs.h, regarding usage in a concurrent environment? (ideally, such notes would annotate individual components of the API in hdfs.h, but if the answer to my question above is, "No", then this could perhaps be a single sentence in the README which affirmatively states that the library is generally safe for concurrent usage without additional/explicit synchronization -- anything would be better than nothing :))



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)