You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/03/14 11:03:01 UTC
[jira] [Commented] (KYLIN-3874) "Convert Cuboid Data to HFile"
failed when HBase and MR use different HDFS clusters
[ https://issues.apache.org/jira/browse/KYLIN-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16792555#comment-16792555 ]
ASF GitHub Bot commented on KYLIN-3874:
---------------------------------------
guangxuCheng commented on pull request #516: KYLIN-3874 "Convert Cuboid Data to HFile" failed when HBase and MR use different HDFS clusters
URL: https://github.com/apache/kylin/pull/516
Detail: https://issues.apache.org/jira/browse/KYLIN-3874
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> "Convert Cuboid Data to HFile" failed when HBase and MR use different HDFS clusters
> -----------------------------------------------------------------------------------
>
> Key: KYLIN-3874
> URL: https://issues.apache.org/jira/browse/KYLIN-3874
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Reporter: Guangxu Cheng
> Assignee: Guangxu Cheng
> Priority: Major
>
> {panel:title=exception}
> 2019-03-14 16:33:45,113 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> ...
> Caused by: java.lang.RuntimeException: Could not find any configured addresses for URI hdfs://{color:#ff0000}hbase-ns-test{color}/kylin/kylin_metadata/kylin-cd4db32e-42b6-44f2-7fa9-ef6c46334249/derived_meta_global/hfile
> at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:116)
> ... 31 more
> {panel}
> {color:#ff0000}hbase-ns-test{color} is the nameservice of HDFS which is used by HBase.
> In our kylin environment, HBase and MR/HIVE use different HDFS clusters.
> {code:java|title=CubeHFileJob.java|borderStyle=solid}
> // use current hbase configuration
> Configuration configuration = new Configuration(HBaseConnection.getCurrentHBaseConfiguration());
> merge(configuration, getConf());
> {code}
> "getConf()" contains only the HDFS configuration of MR, and "configuration" contains the HDFS configuration both HBase and MR. See: [HBaseConnection#addHBaseClusterNNHAConfiguration()|https://github.com/apache/kylin/blob/d23d2bddab3263d39059492b91f71d8091198057/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseConnection.java#L205]
> So "merge(configuration, getConf())" will lead to HBASE's HDFS configuration to be covered. Mainly parameter "dfs.nameservices"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)