You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by "Guangxu Cheng (JIRA)" <ji...@apache.org> on 2019/03/14 10:19:00 UTC
[jira] [Created] (KYLIN-3874) "Convert Cuboid Data to HFile" failed
when HBase and MR use different HDFS clusters
Guangxu Cheng created KYLIN-3874:
------------------------------------
Summary: "Convert Cuboid Data to HFile" failed when HBase and MR use different HDFS clusters
Key: KYLIN-3874
URL: https://issues.apache.org/jira/browse/KYLIN-3874
Project: Kylin
Issue Type: Bug
Components: Job Engine
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
{panel:title=exception}
2019-03-14 16:33:45,113 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
...
Caused by: java.lang.RuntimeException: Could not find any configured addresses for URI hdfs://{color:#ff0000}hbase-ns-test{color}/kylin/kylin_metadata/kylin-cd4db32e-42b6-44f2-7fa9-ef6c46334249/derived_meta_global/hfile
at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:116)
... 31 more
{panel}
{color:#ff0000}hbase-ns-test{color} is the nameservice of HDFS which is used by HBase.
In our kylin environment, HBase and MR/HIVE use different HDFS clusters.
{code:java|title=CubeHFileJob.java|borderStyle=solid}
// use current hbase configuration
Configuration configuration = new Configuration(HBaseConnection.getCurrentHBaseConfiguration());
merge(configuration, getConf());
{code}
"getConf()" contains only the HDFS configuration of MR, and "configuration" contains the HDFS configuration both HBase and MR. See: [HBaseConnection#addHBaseClusterNNHAConfiguration()|https://github.com/apache/kylin/blob/d23d2bddab3263d39059492b91f71d8091198057/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseConnection.java#L205]
So "merge(configuration, getConf())" will lead to HBASE's HDFS configuration to be covered. Mainly parameter "dfs.nameservices"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)