You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by grgrzybek <gi...@git.apache.org> on 2014/03/12 10:11:47 UTC

[GitHub] camel pull request: [CAMEL-7249] camel-hdfs2 component

GitHub user grgrzybek opened a pull request:

    https://github.com/apache/camel/pull/110

    [CAMEL-7249] camel-hdfs2 component

    Finally I've managed to adapt camel-hdfs component to work with hadoop-2.3.0 libraries.
    Thanks to Jean-Baptiste Onofré for providing hadoop-client-2.3.0 OSGi bundle. It'd be hard to use OSGi bundle per hadoop-*.jar artifact because of split packages. So aggregate `hadoop-client` is ok.
    
    Information:
    * Generally camel-hdfs2 component is repackaged version of camel-hdfs component with Maven dependencies switched to hadoop-common:2.3.0 and hadoop-hdfs:2.3.0
    * Some deprecated API usage was replaced by 2.3.0 equivalents
    * This component works in OSGi environment (e.g., inside Apache Karaf) both as classic bundle and blueprint definition
    * There are some quirks related to OSGi:
    * - Hadoop uses `java.util.ServiceLocator` to find filesystem implementations, so some static code initialization must be performed to replace this mechanism (works both in pure OSGi and Blueprint container)
    * - `Import-Package` is not just a list generated by *maven-bundle-plugin* - it must be adjusted to contain all packages reachable by reflection (e.g. `org.apache.hadoop.hdfs*`)
    * - I didn't touch some bundle versions in parent POM and instead I used hardcoded versions for example in AVRO in `platforms/karaf/features/src/main/resources/features.xml`, as the one used in parent POM requires OSGi package `sun.misc`
    * - I had to tweak `org.osgi.framework.system.packages.extra` property in `camel-itest-osgi/**/itest/karaf/config.properties` to contain Xerces packages
    * The documentation must be added for new component

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/grgrzybek/camel CAMEL-7249

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/camel/pull/110.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #110
    
----
commit af7661abb98b85658c35974bb914aae6602365a4
Author: Grzegorz Grzybek <gr...@gmail.com>
Date:   2014-03-10T14:05:43Z

    [CAMEL-7249] Working version of camel-hdfs2 component
    
    * All unit tests works fine.
    * Usage of all deprecated APIs changed.
    * hdfs://localhost tests work (@Ignored - they require external hdfs
    system)

commit 0b9b3531e4a4bfe82492e3037eb36381ad0b1865
Author: Grzegorz Grzybek <gr...@gmail.com>
Date:   2014-03-11T18:29:30Z

    [CAMEL-7249] Working OSGi tests for camel-hdfs2
    
    There are some issues with this implementation:
    * Import-Packages for `camel-hdfs2` are not just the ones generated by
      maven-bundle plugin. They must contain some packages reachable only by
      reflection (different FileSystem implementations)
    * There is problem with using java.util.ServiceLoader inside OSGi and Hadoop 2
      uses ServiceLoader to discover different FileSystem implementations. For OSGi
      (especially blueprint container) we have to use a hack to populate STATIC
      cache
    * `camel-hdfs2` feature contains some hardcoded versions and for e.g., AVRO a
      downgraded one (1.7.5_1 requires `sun.misc` package - this changed however
      since SMX4-1709)
    * org.osgi.framework.system.packages.extra property in
      camel-itest-osgi/**/itest/karaf/config.properties was added with xerces
      packages
    * documentation has to be added describing hadoop2 specific configuration inside
      OSGi - this changed since hadoop 1.2.1 (using java.util.ServiceLoader)

commit 2131dce4d8df10ec207d74f160e926dd74f068c5
Author: Grzegorz Grzybek <gr...@gmail.com>
Date:   2014-03-12T07:05:36Z

    [CAMEL-7249] Working camel-itest-karaf for camel-hdfs2

commit cd101646c688206f73adac68352d14d264287b30
Author: Grzegorz Grzybek <gr...@gmail.com>
Date:   2014-03-12T08:54:29Z

    [CAMEL-7249] Working blueprint tests
    
    All itests now work both with LOCAL and HDFS filesystems

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] camel pull request: [CAMEL-7249] camel-hdfs2 component

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/camel/pull/110


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---