[jira] [Commented] (HADOOP-11127) Improve versioning and compatibility support in native library for downstream hadoop-common users.

["Chris Nauroth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-11127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14147059#comment-14147059 ] 

Chris Nauroth commented on HADOOP-11127:
----------------------------------------

This initially came up during discussion of HADOOP-11064.  It would be good for anyone interested to catch up on the full discussion there.  This is a summary of where we got in the discussion before switching gears in that jira to focus on a quick interim fix.

After further discussion, the following is a summary of potential solutions to this problem:
1. Freeze the libhadoop.so API forever.
2. Library versioning plus maintaining a library on the servers for each supported release.
3. Bundle the .dll or .so file inside a jar somehow so that YARN / Slider can distribute it.

#1. Advantages:
* ???

#1. Disadvantages:
* We're unable to improve libhadoop.so in the future.
* There will be puzzling interactions when mixing and matching versions. "New" bugs in libhadoop.so will show up with old hadoop releases, causing confusion in bug trackers.
We don't have any way of enforcing C API stability. Jenkins doesn't check for it, most Java programmers don't know how to achieve it.
* There is still no ability for applications using new Hadoop versions to make use of old libhadoop.so versions, unless we adopt an even worse compatibility policy that nothing new can be added to libhadoop.so.
* Given all of the above, this option seems to be off the table.

#2. Advantages:
* Simple to implement.
* There's already a patch that implements it.
* We want libhadoop.so library versioning anyway, even if we later adopt another solution in addition to this

#2. Disadvantages:
* Admins using Slider / YARN will need to ensure that the appropriate versions of libhadoop are present on the server.

#3. Advantages:
* "Cleanest" solution, since it allows us to reuse YARN's existing distribution mechanisms.

#3. Disadvantages:
* There are technical challenges to bundling a library in a jar that we haven't yet tackled.

> Improve versioning and compatibility support in native library for downstream hadoop-common users.
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-11127
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11127
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>            Reporter: Chris Nauroth
>
> There is no compatibility policy enforced on the JNI function signatures implemented in the native library.  This library typically is deployed to all nodes in a cluster, built from a specific source code version.  However, downstream applications that want to run in that cluster might choose to bundle a hadoop-common jar at a different version.  Since there is no compatibility policy, this can cause link errors at runtime when the native function signatures expected by hadoop-common.jar do not exist in libhadoop.so/hadoop.dll.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)