You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2019/10/11 07:32:00 UTC
[jira] [Commented] (KYLIN-4190) hiveproducer write() function throw
exception because hive mertics table location path prefix is different with
defaut fs when hdfs uses router-based federation
[ https://issues.apache.org/jira/browse/KYLIN-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949215#comment-16949215 ]
ASF GitHub Bot commented on KYLIN-4190:
---------------------------------------
bigxiaochu commented on pull request #875: KYLIN-4190 hiveproducer write() function throw exception because hive…
URL: https://github.com/apache/kylin/pull/875
https://issues.apache.org/jira/browse/KYLIN-4190
hiveproducer write() function throw exception because hive mertics table location path prefix is different with defaut fs when hdfs uses router-based federation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> hiveproducer write() function throw exception because hive mertics table location path prefix is different with defaut fs when hdfs uses router-based federation
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KYLIN-4190
> URL: https://issues.apache.org/jira/browse/KYLIN-4190
> Project: Kylin
> Issue Type: Bug
> Components: Metrics
> Affects Versions: v3.0.0-alpha
> Reporter: chuxiao
> Priority: Critical
> Attachments: KYLIN-4190.master.001.patch
>
>
> Our hdfs cluster uses router-based federation（[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html]).
> Opening the system cube configuration, hiveproducer write() function throw exception :
> {code:java}
> //代码占位符
> ERROR [metrics-blocking-reservoir-scheduler-0] hive.HiveReservoirReporter:119 : Wrong FS: hdfs://DClusterNmg4/user/kylin/hive/hive_metrics_job_exception_qa/kday_date=2019-09-04, expected: hdfs://difed
> java.lang.IllegalArgumentException: Wrong FS: hdfs://DClusterNmg4/user/kylin/hive/hive_metrics_job_exception_qa/kday_date=2019-09-04, expected: hdfs://difed
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:717)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:197)
> at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:109)
> at org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1390)
> at org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1386)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1402)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1494)
> at org.apache.kylin.metrics.lib.impl.hive.HiveProducer.write(HiveProducer.java:137)
> at org.apache.kylin.metrics.lib.impl.hive.HiveProducer.send(HiveProducer.java:122)
> at org.apache.kylin.metrics.lib.impl.hive.HiveReservoirReporter$HiveReservoirListener.onRecordUpdate(HiveReservoirReporter.java:117)
> at org.apache.kylin.metrics.lib.impl.BlockingReservoir.notifyListenerOfUpdatedRecord(BlockingReservoir.java:105)
> at org.apache.kylin.metrics.lib.impl.BlockingReservoir.onRecordUpdate(BlockingReservoir.java:93)
> at org.apache.kylin.metrics.lib.impl.BlockingReservoir.access$300(BlockingReservoir.java:33)
> at org.apache.kylin.metrics.lib.impl.BlockingReservoir$ReporterRunnable.run(BlockingReservoir.java:152)
> at java.lang.Thread.run(Thread.java:745)
> {code}
>
> This is because the default router namespace is hdfs://difed, and the actual federation namespaces are the hdfs://DClusterNmg4, the hdfs://DClusterNmg1, and the hdfs://DClusterNmg2...
> So fs.defaultFS in core-sie.xml is hdfs ://difed, But this hive table location path is hdfs://DClusterNmg4/user/... . Then defaultFs.exists(hiveLocationPath) throw exception.
> So we need to check if the prefix is same. If defaut fs is not a prefix of hive table location path, use location path get a new filesystem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)