You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Dmytro Sen <ds...@hortonworks.com> on 2014/10/16 17:19:15 UTC
Review Request 26810: Flume agent on Ambari uses the default Java on
machine
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26810/
-----------------------------------------------------------
Review request for Ambari and Vitalyi Brodetskyi.
Bugs: AMBARI-7812
https://issues.apache.org/jira/browse/AMBARI-7812
Repository: ambari
Description
-------
When running Flume Agent on Ambari installed cluster, the command aborts with OutOfMemory, here is a sample run:
/usr/hdp/current/flume-client/bin/flume-ng agent -n agent -c /usr/hdp/current/flume-client/conf -f /grid/0/hadoopqe/tests/flume/conf/avro-memory-file_roll.properties -Dflume.root.logger=DEBUG,console
Warning: JAVA_HOME is not set!
Info: Including Hadoop libraries found via (/usr/bin/hadoop) for HDFS access
Info: Excluding /usr/hdp/2.2.0.0-917/hadoop/lib/slf4j-api-1.7.5.jar from classpath
Info: Excluding /usr/hdp/2.2.0.0-917/hadoop/lib/slf4j-log4j12-1.7.5.jar from classpath
Info: Including HBASE libraries found via (/usr/bin/hbase) for HBASE access
Info: Excluding /usr/hdp/2.2.0.0-917/hbase/lib/slf4j-api-1.6.4.jar from classpath
Info: Excluding /usr/hdp/2.2.0.0-917/hadoop/lib/slf4j-api-1.7.5.jar from classpath
Info: Excluding /usr/hdp/2.2.0.0-917/hadoop/lib/slf4j-log4j12-1.7.5.jar from classpath
Info: Excluding /usr/hdp/2.2.0.0-917/hadoop/lib/slf4j-api-1.7.5.jar from classpath
Info: Excluding /usr/hdp/2.2.0.0-917/hadoop/lib/slf4j-log4j12-1.7.5.jar from classpath
Info: Excluding /usr/hdp/2.2.0.0-917/zookeeper/lib/slf4j-api-1.6.1.jar from classpath
Info: Excluding /usr/hdp/2.2.0.0-917/zookeeper/lib/slf4j-log4j12-1.6.1.jar from classpath
Info: Including Hive libraries found via () for Hive access
+ exec /usr/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '...' -n agent -f /grid/0/hadoopqe/tests/flume/conf/avro-memory-file_roll.properties
GC Warning: Out of Memory! Returning NIL!
Exception during runtime initialization
GC Warning: Out of Memory! Returning NIL!
I think the issue here is as we see in the console output that Flume Agent complains that no JAVA_HOME is set so it picks up the java in the default path which is /usr/bin/java.
# /usr/bin/java -version
java version "1.5.0"
gij (GNU libgcj) version 4.4.7 20120313 (Red Hat 4.4.7-4)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
As we see that it uses some 1.5.0 release of Java. We should have used the installed JDK.
# /usr/jdk64/jdk1.7.0_67/bin/java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
Basically we should set JAVA_HOME in $FLUME_CONF_DIR/flume-env.sh. In gsInstaller install we append the following line to flume-env.sh.
export JAVA_HOME=<Installed Java Home>
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/FLUME/configuration/flume-env.xml 7b11bde
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/FLUME/package/scripts/flume.py 45d0c8f
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/FLUME/package/scripts/params.py e0bec7b
ambari-server/src/test/python/stacks/2.0.6/FLUME/test_flume.py 58f3759
ambari-server/src/test/python/stacks/2.0.6/configs/default.json b8ad8ff
ambari-server/src/test/python/stacks/2.0.6/configs/default.non_gmetad_host.json 24397b0
ambari-server/src/test/python/stacks/2.0.6/configs/flume_target.json 853c73c
ambari-server/src/test/python/stacks/2.0.6/configs/secured.json fa6f0b6
Diff: https://reviews.apache.org/r/26810/diff/
Testing
-------
----------------------------------------------------------------------
Ran 234 tests in 3.391s
OK
----------------------------------------------------------------------
Total run:644
Total errors:0
Total failures:0
OK
Thanks,
Dmytro Sen
Re: Review Request 26810: Flume agent on Ambari uses the default Java
on machine
Posted by Vitalyi Brodetskyi <vb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26810/#review56922
-----------------------------------------------------------
Ship it!
Ship It!
- Vitalyi Brodetskyi
On Жов. 16, 2014, 3:19 після полудня, Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26810/
> -----------------------------------------------------------
>
> (Updated Жов. 16, 2014, 3:19 після полудня)
>
>
> Review request for Ambari and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-7812
> https://issues.apache.org/jira/browse/AMBARI-7812
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When running Flume Agent on Ambari installed cluster, the command aborts with OutOfMemory, here is a sample run:
> /usr/hdp/current/flume-client/bin/flume-ng agent -n agent -c /usr/hdp/current/flume-client/conf -f /grid/0/hadoopqe/tests/flume/conf/avro-memory-file_roll.properties -Dflume.root.logger=DEBUG,console
> Warning: JAVA_HOME is not set!
> Info: Including Hadoop libraries found via (/usr/bin/hadoop) for HDFS access
> Info: Excluding /usr/hdp/2.2.0.0-917/hadoop/lib/slf4j-api-1.7.5.jar from classpath
> Info: Excluding /usr/hdp/2.2.0.0-917/hadoop/lib/slf4j-log4j12-1.7.5.jar from classpath
> Info: Including HBASE libraries found via (/usr/bin/hbase) for HBASE access
> Info: Excluding /usr/hdp/2.2.0.0-917/hbase/lib/slf4j-api-1.6.4.jar from classpath
> Info: Excluding /usr/hdp/2.2.0.0-917/hadoop/lib/slf4j-api-1.7.5.jar from classpath
> Info: Excluding /usr/hdp/2.2.0.0-917/hadoop/lib/slf4j-log4j12-1.7.5.jar from classpath
> Info: Excluding /usr/hdp/2.2.0.0-917/hadoop/lib/slf4j-api-1.7.5.jar from classpath
> Info: Excluding /usr/hdp/2.2.0.0-917/hadoop/lib/slf4j-log4j12-1.7.5.jar from classpath
> Info: Excluding /usr/hdp/2.2.0.0-917/zookeeper/lib/slf4j-api-1.6.1.jar from classpath
> Info: Excluding /usr/hdp/2.2.0.0-917/zookeeper/lib/slf4j-log4j12-1.6.1.jar from classpath
> Info: Including Hive libraries found via () for Hive access
> + exec /usr/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '...' -n agent -f /grid/0/hadoopqe/tests/flume/conf/avro-memory-file_roll.properties
> GC Warning: Out of Memory! Returning NIL!
> Exception during runtime initialization
> GC Warning: Out of Memory! Returning NIL!
> I think the issue here is as we see in the console output that Flume Agent complains that no JAVA_HOME is set so it picks up the java in the default path which is /usr/bin/java.
> # /usr/bin/java -version
> java version "1.5.0"
> gij (GNU libgcj) version 4.4.7 20120313 (Red Hat 4.4.7-4)
>
> Copyright (C) 2007 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions. There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> As we see that it uses some 1.5.0 release of Java. We should have used the installed JDK.
> # /usr/jdk64/jdk1.7.0_67/bin/java -version
> java version "1.7.0_67"
> Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
> Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
> Basically we should set JAVA_HOME in $FLUME_CONF_DIR/flume-env.sh. In gsInstaller install we append the following line to flume-env.sh.
> export JAVA_HOME=<Installed Java Home>
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/FLUME/configuration/flume-env.xml 7b11bde
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/FLUME/package/scripts/flume.py 45d0c8f
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/FLUME/package/scripts/params.py e0bec7b
> ambari-server/src/test/python/stacks/2.0.6/FLUME/test_flume.py 58f3759
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json b8ad8ff
> ambari-server/src/test/python/stacks/2.0.6/configs/default.non_gmetad_host.json 24397b0
> ambari-server/src/test/python/stacks/2.0.6/configs/flume_target.json 853c73c
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json fa6f0b6
>
> Diff: https://reviews.apache.org/r/26810/diff/
>
>
> Testing
> -------
>
> ----------------------------------------------------------------------
> Ran 234 tests in 3.391s
>
> OK
> ----------------------------------------------------------------------
> Total run:644
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Dmytro Sen
>
>