You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@fluo.apache.org by GitBox <gi...@apache.org> on 2020/07/09 22:39:34 UTC
[GitHub] [fluo] arvindshmicrosoft opened a new pull request #1101: Add Hadoop dirs to classpath and native lib path
arvindshmicrosoft opened a new pull request #1101:
URL: https://github.com/apache/fluo/pull/1101
* Adds the $HADOOP_HOME/etc/hadoop directory to Fluo classpath, so that
hdfs-site.xml can be located and loaded
* Adds the Hadoop native library directory to the LD_LIBRARY_PATH /
DYLD_LIBRARY_PATH for the OS
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo] arvindshmicrosoft commented on pull request #1101: Add Hadoop dirs to classpath and native lib path
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft commented on pull request #1101:
URL: https://github.com/apache/fluo/pull/1101#issuecomment-656386328
For context, the original issue encountered (and which is fixed by this PR) is that when Fluo is configured to use a highly-available (HA) HDFS namespace as the DFS root, fluo-env.sh does not add the Hadoop conf folder to the classpath. This leads to a error as shown below. In this case, `muchoshacluster` was the HA HDFS namespace configured.
```
2020-07-09 22:30:39,322 [client.FluoAdminImpl] ERROR: Failed to create DFS directory hdfs://muchoshacluster/fluo/webindex/lib/accumulo
2020-07-09 22:30:39,425 [zookeeper.ZooCache] WARN : Unhandled: WatchedEvent state:Closed type:None path:null
Exception in thread "main" java.lang.IllegalStateException: java.lang.IllegalArgumentException: java.net.UnknownHostException: muchoshacluster
at org.apache.fluo.core.client.FluoAdminImpl.copyJarsToDfs(FluoAdminImpl.java:393)
at org.apache.fluo.core.client.FluoAdminImpl.initialize(FluoAdminImpl.java:178)
...
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: muchoshacluster
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:447)
at org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:139)
...
at org.apache.fluo.core.client.FluoAdminImpl.copyJarsToDfs(FluoAdminImpl.java:382)
... 5 more
Caused by: java.net.UnknownHostException: muchoshacluster
```
Regular Hadoop commands worked fine on that node because Hadoop sets up its classpath to correctly load hdfs-site.xml from the conf folder.
Also, the Hadoop native library is not on the system library load path, leading to the below warning:
```
2020-07-09 22:30:39,076 [util.NativeCodeLoader] WARN : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo] arvindshmicrosoft merged pull request #1101: Add Hadoop dirs to classpath and native lib path
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft merged pull request #1101:
URL: https://github.com/apache/fluo/pull/1101
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org