You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Jonathan Hsieh (JIRA)" <ji...@apache.org> on 2011/09/19 01:38:08 UTC
[jira] [Updated] (FLUME-580) Flume needs to be consistent with
autodiscovery of Hadoop compression codecs
[ https://issues.apache.org/jira/browse/FLUME-580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated FLUME-580:
---------------------------------
Assignee: Jonathan Hsieh
Summary: Flume needs to be consistent with autodiscovery of Hadoop compression codecs (was: Flume needs to be consistent with autodiscovery of HadoopLZO codec)
> Flume needs to be consistent with autodiscovery of Hadoop compression codecs
> ----------------------------------------------------------------------------
>
> Key: FLUME-580
> URL: https://issues.apache.org/jira/browse/FLUME-580
> Project: Flume
> Issue Type: Bug
> Reporter: Disabled imported user
> Assignee: Jonathan Hsieh
>
> Currently, if flume notices that LZO is configured in core-site.xml it tries to load it. However, since it doesn't follow the same route that Hadoop does for locating the JAR and corresponding so files it fails to do so. Flume has to leverage the same logic that hadoop has embedded in bin/hadoop:
> .....
> # setup 'java.library.path' for native-hadoop code if necessary
> if [ -d "${HADOOP_HOME}/build/native" -o -d "${HADOOP_HOME}/lib/native" -o -d "${HADOOP_HOME}/sbin" ]; then
> JAVA_PLATFORM=`CLASSPATH=${CLASSPATH} ${JAVA} -Xmx32m ${HADOOP_JAVA_PLATFORM_OPTS} org.apache.hadoop.util.PlatformName | sed -e "s/ /_/g"`
> ...
> so that it can transparently find LZO objects that it tries to autodiscover.
> For the time being, however, the workaround seems to be to tell flume explicitly where to look for LZO via
> adding the following to /usr/lib/flume/bin/flume-env.sh:
> export JAVA_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/
> export FLUME_CLASSPATH=/usr/lib/hadoop-0.20/lib/hadoop-lzo-20101122174751.20101122171345.552b3f9.jar
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira