You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 雨日听风 <49...@qq.com> on 2016/12/20 06:51:41 UTC
Re： Error when #2 Step: Redistribute Flat Hive Table - File does not exist
Thank you!
We checked the yarn and hard disk. But not found any error. Hard disk space and memory and so on is working well.
Last time its error code was "unknownhost clusterB",now in new server env it cant find clusterB(hbase only). but cant find rowCount file.
===
the follow command runs ok:
hdfs dfs -mkdir /young/kylin_test/kylin_metadata_nokia/kylin-678c15ba-5375-4f80-831e-1ae0af8ed576/row_count/tmp
And "ls" cant find file "000000_0" which it said "file does not exist".
------------------ 原始邮件 ------------------
发件人: "Alberto Ramón";<a....@gmail.com>;
发送时间: 2016年12月19日(星期一) 晚上9:13
收件人: "user"<us...@kylin.apache.org>;
主题: Re: Error when #2 Step: Redistribute Flat Hive Table - File does not exist
i think i had this error last nigth :) (go to yarn to find detailed error & find on internet)
in my case was free space less than 10% of hard disk. Check this please
El 19/12/2016 11:35, "雨日听风" <49...@qq.com> escribió:
When I build a cube in kylin1.6, I get error in step #2: Redistribute Flat Hive Table
Please help! Thank you very much!
env: kylin1.6 is in a independent server, and have 2 other server cluster: clusterA(hive only) and clusterB(hbase only).
Error is:
2016-12-19 10:28:00,641 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Compute row count of flat hive table, cmd:
2016-12-19 10:28:00,642 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : hive -e "USE boco;
SET dfs.replication=2;
SET hive.exec.compress.output=true;
SET hive.auto.convert.join.noconditionaltask=true;
SET hive.auto.convert.join.noconditionaltask.size=100000000;
SET mapreduce.output.fileoutputformat.compress.type=BLOCK;
SET mapreduce.job.split.metainfo.maxsize=-1;
SET mapreduce.job.queuename=young;
SET tez.queue.name=young;
set hive.exec.compress.output=false;
set hive.exec.compress.output=false;
INSERT OVERWRITE DIRECTORY '/young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count' SELECT count(*) FROM kylin_intermediate_hbase_in_testCluster_CUBE_f9468805_eabf_4b54_bf2b_182e4c86214a;
"
2016-12-19 10:28:03,277 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : WARNING: Use "yarn jar" to launch YARN applications.
2016-12-19 10:28:04,444 INFO [pool-8-thread-7] execution.AbstractExecutable:36 :
2016-12-19 10:28:04,445 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Logging initialized using configuration in file:/etc/hive/conf/hive-log4j.properties
2016-12-19 10:28:14,700 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : OK
2016-12-19 10:28:14,703 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Time taken: 0.935 seconds
2016-12-19 10:28:15,559 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Query ID = young_20161219102814_a7104fd4-ba83-47fc-ac0b-0c9bef4e1969
2016-12-19 10:28:15,560 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Total jobs = 1
2016-12-19 10:28:15,575 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Launching Job 1 out of 1
2016-12-19 10:28:22,842 INFO [pool-8-thread-7] execution.AbstractExecutable:36 :
2016-12-19 10:28:22,842 INFO [pool-8-thread-7] execution.AbstractExecutable:36 :
2016-12-19 10:28:23,104 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Status: Running (Executing on YARN cluster with App id application_1473415773736_1063281)
2016-12-19 10:28:23,104 INFO [pool-8-thread-7] execution.AbstractExecutable:36 :
2016-12-19 10:28:23,104 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: -/- Reducer 2: 0/1
2016-12-19 10:28:23,307 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1
2016-12-19 10:28:26,363 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1
2016-12-19 10:28:26,567 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0(+1)/2 Reducer 2: 0/1
2016-12-19 10:28:26,596 INFO [pool-7-thread-1] threadpool.DefaultScheduler:118 : Job Fetcher: 1 should running, 1 actual running, 0 ready, 0 already succeed, 3 error, 1 discarded, 0 others
2016-12-19 10:28:26,769 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1
2016-12-19 10:28:29,810 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1
2016-12-19 10:28:30,217 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 1(+1)/2 Reducer 2: 0(+1)/1
2016-12-19 10:28:30,826 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 0(+1)/1
2016-12-19 10:28:31,232 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 1/1
2016-12-19 10:28:31,319 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Moving data to: /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count
2016-12-19 10:28:31,406 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : OK
2016-12-19 10:28:31,454 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Time taken: 16.701 seconds
2016-12-19 10:28:35,074 ERROR [pool-8-thread-7] execution.AbstractExecutable:357 : job:678266c0-ba0e-48b4-bdb5-6e578320375a-01 execute finished with exception
java.io.FileNotFoundException: File does not exist: /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/000000_0
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1820)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1791)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1704)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:587)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1242)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1227)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1215)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:303)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:269)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:261)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1540)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:303)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:299)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.readRowCountFromFile(HiveMRInput.java:294)
at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.doWork(HiveMRInput.java:337)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/000000_0
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1820)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1791)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1704)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:587)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy25.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy26.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1240)
... 20 more
2016-12-19 10:28:35,078 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 : updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
2016-12-19 10:28:35,082 DEBUG [pool-8-thread-7] hbase.HBaseResourceStore:262 : Update row /execute_output/678266c0-ba0e-48b4-bdb5-6e578320375a-01 from oldTs: 1482114480638, to newTs: 1482114515078, operation result: true
2016-12-19 10:28:35,085 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 : updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
Re: Error when #2 Step: Redistribute Flat Hive Table - File does not exist
Posted by ShaoFeng Shi <sh...@apache.org>.
BTW: KYLIN-2165 will handle this in a more elegant way, it will be released
in next version;
2016-12-22 9:21 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> "java.io.FileNotFoundException: File does not exist:
> /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-
> 48b4-bdb5-6e578320375a/row_count/000000_0"
>
> Seems be the issue in https://issues.apache.org/jira/browse/KYLIN-2159
> It should be fixed in 1.6.0; could you please check whether there is any
> file in the folder?
>
> hadoop fs -ls /young/kylin_test/kylin_metadata_test/kylin-678266c0-
> ba0e-48b4-bdb5-6e578320375a/row_count/
>
> 2016-12-20 15:23 GMT+08:00 Alberto Ramón <a....@gmail.com>:
>
>> other idea:
>> Can be a problem with permissions?: the user that execute Kylin can't
>> read data generated by YARN
>> check if Kylin user can read your folder /young/kylin_test/
>> Which Hadoop user are executing Kylin?
>>
>> (no more ideas, Good Luck)
>>
>> 2016-12-20 7:51 GMT+01:00 雨日听风 <49...@qq.com>:
>>
>>> Thank you!
>>> We checked the yarn and hard disk. But not found any error. Hard disk
>>> space and memory and so on is working well.
>>> Last time its error code was "unknownhost clusterB",now in new server
>>> env it cant find clusterB(hbase only). but cant find rowCount file.
>>> ===
>>> the follow command runs ok:
>>> hdfs dfs -mkdir /young/kylin_test/kylin_metadata_nokia/kylin
>>> -678c15ba-5375-4f80-831e-1ae0af8ed576/row_count/tmp
>>> And "ls" cant find file "000000_0" which it said "file does not exist".
>>>
>>> ------------------ 原始邮件 ------------------
>>> *发件人:* "Alberto Ramón";<a....@gmail.com>;
>>> *发送时间:* 2016年12月19日(星期一) 晚上9:13
>>> *收件人:* "user"<us...@kylin.apache.org>;
>>> *主题:* Re: Error when #2 Step: Redistribute Flat Hive Table - File does
>>> not exist
>>>
>>> i think i had this error last nigth :)
>>> (go to yarn to find detailed error & find on internet)
>>> in my case was free space less than 10% of hard disk. Check this please
>>>
>>> El 19/12/2016 11:35, "雨日听风" <49...@qq.com> escribió:
>>>
>>>> When I build a cube in kylin1.6, I get error in step #2: Redistribute
>>>> Flat Hive Table
>>>>
>>>> Please help! Thank you very much!
>>>>
>>>> env: kylin1.6 is in a independent server, and have 2 other server
>>>> cluster: clusterA(hive only) and clusterB(hbase only).
>>>> Error is:
>>>>
>>>> 2016-12-19 10:28:00,641 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Compute row count of flat hive
>>>> table, cmd:
>>>> 2016-12-19 10:28:00,642 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : hive -e "USE boco;
>>>> SET dfs.replication=2;
>>>> SET hive.exec.compress.output=true;
>>>> SET hive.auto.convert.join.noconditionaltask=true;
>>>> SET hive.auto.convert.join.noconditionaltask.size=100000000;
>>>> SET mapreduce.output.fileoutputformat.compress.type=BLOCK;
>>>> SET mapreduce.job.split.metainfo.maxsize=-1;
>>>> SET mapreduce.job.queuename=young;
>>>> SET tez.queue.name=young;
>>>>
>>>> set hive.exec.compress.output=false;
>>>>
>>>> set hive.exec.compress.output=false;
>>>> INSERT OVERWRITE DIRECTORY '/young/kylin_test/kylin_metad
>>>> ata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count' SELECT
>>>> count(*) FROM kylin_intermediate_hbase_in_te
>>>> stCluster_CUBE_f9468805_eabf_4b54_bf2b_182e4c86214a;
>>>>
>>>> "
>>>> 2016-12-19 10:28:03,277 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : WARNING: Use "yarn jar" to launch
>>>> YARN applications.
>>>> 2016-12-19 10:28:04,444 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 :
>>>> 2016-12-19 10:28:04,445 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Logging initialized using
>>>> configuration in file:/etc/hive/conf/hive-log4j.properties
>>>> 2016-12-19 10:28:14,700 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : OK
>>>> 2016-12-19 10:28:14,703 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Time taken: 0.935 seconds
>>>> 2016-12-19 10:28:15,559 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Query ID =
>>>> young_20161219102814_a7104fd4-ba83-47fc-ac0b-0c9bef4e1969
>>>> 2016-12-19 10:28:15,560 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Total jobs = 1
>>>> 2016-12-19 10:28:15,575 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Launching Job 1 out of 1
>>>> 2016-12-19 10:28:22,842 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 :
>>>> 2016-12-19 10:28:22,842 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 :
>>>> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Status: Running (Executing on YARN
>>>> cluster with App id application_1473415773736_1063281)
>>>> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 :
>>>> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Map 1: -/- Reducer 2: 0/1
>>>> 2016-12-19 10:28:23,307 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1
>>>> 2016-12-19 10:28:26,363 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1
>>>> 2016-12-19 10:28:26,567 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Map 1: 0(+1)/2 Reducer 2: 0/1
>>>> 2016-12-19 10:28:26,596 INFO [pool-7-thread-1]
>>>> threadpool.DefaultScheduler:118 : Job Fetcher: 1 should running, 1
>>>> actual running, 0 ready, 0 already succeed, 3 error, 1 discarded, 0 others
>>>> 2016-12-19 10:28:26,769 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1
>>>> 2016-12-19 10:28:29,810 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1
>>>> 2016-12-19 10:28:30,217 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Map 1: 1(+1)/2 Reducer 2: 0(+1)/1
>>>> 2016-12-19 10:28:30,826 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 0(+1)/1
>>>> 2016-12-19 10:28:31,232 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 1/1
>>>> 2016-12-19 10:28:31,319 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Moving data to:
>>>> /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48
>>>> b4-bdb5-6e578320375a/row_count
>>>> 2016-12-19 10:28:31,406 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : OK
>>>> 2016-12-19 10:28:31,454 INFO [pool-8-thread-7]
>>>> execution.AbstractExecutable:36 : Time taken: 16.701 seconds
>>>> 2016-12-19 10:28:35,074 ERROR [pool-8-thread-7]
>>>> execution.AbstractExecutable:357 : job:678266c0-ba0e-48b4-bdb5-6e578320375a-01
>>>> execute finished with exception
>>>> java.io.FileNotFoundException: File does not exist:
>>>> /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48
>>>> b4-bdb5-6e578320375a/row_count/000000_0
>>>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>>>> deFile.java:71)
>>>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>>>> deFile.java:61)
>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>>>> LocationsInt(FSNamesystem.java:1820)
>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>>>> Locations(FSNamesystem.java:1791)
>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>>>> Locations(FSNamesystem.java:1704)
>>>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.get
>>>> BlockLocations(NameNodeRpcServer.java:587)
>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ
>>>> erSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolS
>>>> erverSideTranslatorPB.java:365)
>>>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol
>>>> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
>>>> enodeProtocolProtos.java)
>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn
>>>> voker.call(ProtobufRpcEngine.java:616)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>>>> upInformation.java:1657)
>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)
>>>>
>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>> Method)
>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Native
>>>> ConstructorAccessorImpl.java:57)
>>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(De
>>>> legatingConstructorAccessorImpl.java:45)
>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>> at org.apache.hadoop.ipc.RemoteException.instantiateException(R
>>>> emoteException.java:106)
>>>> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(
>>>> RemoteException.java:73)
>>>> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSCl
>>>> ient.java:1242)
>>>> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.
>>>> java:1227)
>>>> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.
>>>> java:1215)
>>>> at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndG
>>>> etLastBlockLength(DFSInputStream.java:303)
>>>> at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStrea
>>>> m.java:269)
>>>> at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.
>>>> java:261)
>>>> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1540)
>>>> at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(Distri
>>>> butedFileSystem.java:303)
>>>> at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(Distri
>>>> butedFileSystem.java:299)
>>>> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSyst
>>>> emLinkResolver.java:81)
>>>> at org.apache.hadoop.hdfs.DistributedFileSystem.open(Distribute
>>>> dFileSystem.java:299)
>>>> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
>>>> at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiv
>>>> eTableStep.readRowCountFromFile(HiveMRInput.java:294)
>>>> at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiv
>>>> eTableStep.doWork(HiveMRInput.java:337)
>>>> at org.apache.kylin.job.execution.AbstractExecutable.execute(Ab
>>>> stractExecutable.java:113)
>>>> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWo
>>>> rk(DefaultChainedExecutable.java:57)
>>>> at org.apache.kylin.job.execution.AbstractExecutable.execute(Ab
>>>> stractExecutable.java:113)
>>>> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRun
>>>> ner.run(DefaultScheduler.java:136)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>>> Executor.java:1145)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>>> lExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException):
>>>> File does not exist: /young/kylin_test/kylin_metada
>>>> ta_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/000000_0
>>>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>>>> deFile.java:71)
>>>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>>>> deFile.java:61)
>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>>>> LocationsInt(FSNamesystem.java:1820)
>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>>>> Locations(FSNamesystem.java:1791)
>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>>>> Locations(FSNamesystem.java:1704)
>>>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.get
>>>> BlockLocations(NameNodeRpcServer.java:587)
>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ
>>>> erSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolS
>>>> erverSideTranslatorPB.java:365)
>>>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol
>>>> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
>>>> enodeProtocolProtos.java)
>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn
>>>> voker.call(ProtobufRpcEngine.java:616)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>>>> upInformation.java:1657)
>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)
>>>>
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1358)
>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(Proto
>>>> bufRpcEngine.java:229)
>>>> at com.sun.proxy.$Proxy25.getBlockLocations(Unknown Source)
>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran
>>>> slatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>>> ssorImpl.java:57)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>> thodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth
>>>> od(RetryInvocationHandler.java:252)
>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret
>>>> ryInvocationHandler.java:104)
>>>> at com.sun.proxy.$Proxy26.getBlockLocations(Unknown Source)
>>>> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSCl
>>>> ient.java:1240)
>>>> ... 20 more
>>>> 2016-12-19 10:28:35,078 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 :
>>>> updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
>>>> 2016-12-19 10:28:35,082 DEBUG [pool-8-thread-7]
>>>> hbase.HBaseResourceStore:262 : Update row /execute_output/678266c0-ba0e-48b4-bdb5-6e578320375a-01
>>>> from oldTs: 1482114480638, to newTs: 1482114515078, operation result: true
>>>> 2016-12-19 10:28:35,085 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 :
>>>> updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
>>>>
>>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Error when #2 Step: Redistribute Flat Hive Table - File does not exist
Posted by ShaoFeng Shi <sh...@apache.org>.
"java.io.FileNotFoundException: File does not exist:
/young/kylin_test/kylin_metadata_test/kylin-678266c0-
ba0e-48b4-bdb5-6e578320375a/row_count/000000_0"
Seems be the issue in https://issues.apache.org/jira/browse/KYLIN-2159
It should be fixed in 1.6.0; could you please check whether there is any
file in the folder?
hadoop fs -ls
/young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/
2016-12-20 15:23 GMT+08:00 Alberto Ramón <a....@gmail.com>:
> other idea:
> Can be a problem with permissions?: the user that execute Kylin can't
> read data generated by YARN
> check if Kylin user can read your folder /young/kylin_test/
> Which Hadoop user are executing Kylin?
>
> (no more ideas, Good Luck)
>
> 2016-12-20 7:51 GMT+01:00 雨日听风 <49...@qq.com>:
>
>> Thank you!
>> We checked the yarn and hard disk. But not found any error. Hard disk
>> space and memory and so on is working well.
>> Last time its error code was "unknownhost clusterB",now in new server env
>> it cant find clusterB(hbase only). but cant find rowCount file.
>> ===
>> the follow command runs ok:
>> hdfs dfs -mkdir /young/kylin_test/kylin_metadata_nokia/kylin
>> -678c15ba-5375-4f80-831e-1ae0af8ed576/row_count/tmp
>> And "ls" cant find file "000000_0" which it said "file does not exist".
>>
>> ------------------ 原始邮件 ------------------
>> *发件人:* "Alberto Ramón";<a....@gmail.com>;
>> *发送时间:* 2016年12月19日(星期一) 晚上9:13
>> *收件人:* "user"<us...@kylin.apache.org>;
>> *主题:* Re: Error when #2 Step: Redistribute Flat Hive Table - File does
>> not exist
>>
>> i think i had this error last nigth :)
>> (go to yarn to find detailed error & find on internet)
>> in my case was free space less than 10% of hard disk. Check this please
>>
>> El 19/12/2016 11:35, "雨日听风" <49...@qq.com> escribió:
>>
>>> When I build a cube in kylin1.6, I get error in step #2: Redistribute
>>> Flat Hive Table
>>>
>>> Please help! Thank you very much!
>>>
>>> env: kylin1.6 is in a independent server, and have 2 other server
>>> cluster: clusterA(hive only) and clusterB(hbase only).
>>> Error is:
>>>
>>> 2016-12-19 10:28:00,641 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Compute row count of flat hive table,
>>> cmd:
>>> 2016-12-19 10:28:00,642 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : hive -e "USE boco;
>>> SET dfs.replication=2;
>>> SET hive.exec.compress.output=true;
>>> SET hive.auto.convert.join.noconditionaltask=true;
>>> SET hive.auto.convert.join.noconditionaltask.size=100000000;
>>> SET mapreduce.output.fileoutputformat.compress.type=BLOCK;
>>> SET mapreduce.job.split.metainfo.maxsize=-1;
>>> SET mapreduce.job.queuename=young;
>>> SET tez.queue.name=young;
>>>
>>> set hive.exec.compress.output=false;
>>>
>>> set hive.exec.compress.output=false;
>>> INSERT OVERWRITE DIRECTORY '/young/kylin_test/kylin_metad
>>> ata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count' SELECT
>>> count(*) FROM kylin_intermediate_hbase_in_te
>>> stCluster_CUBE_f9468805_eabf_4b54_bf2b_182e4c86214a;
>>>
>>> "
>>> 2016-12-19 10:28:03,277 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : WARNING: Use "yarn jar" to launch
>>> YARN applications.
>>> 2016-12-19 10:28:04,444 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 :
>>> 2016-12-19 10:28:04,445 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Logging initialized using
>>> configuration in file:/etc/hive/conf/hive-log4j.properties
>>> 2016-12-19 10:28:14,700 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : OK
>>> 2016-12-19 10:28:14,703 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Time taken: 0.935 seconds
>>> 2016-12-19 10:28:15,559 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Query ID =
>>> young_20161219102814_a7104fd4-ba83-47fc-ac0b-0c9bef4e1969
>>> 2016-12-19 10:28:15,560 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Total jobs = 1
>>> 2016-12-19 10:28:15,575 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Launching Job 1 out of 1
>>> 2016-12-19 10:28:22,842 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 :
>>> 2016-12-19 10:28:22,842 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 :
>>> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Status: Running (Executing on YARN
>>> cluster with App id application_1473415773736_1063281)
>>> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 :
>>> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Map 1: -/- Reducer 2: 0/1
>>> 2016-12-19 10:28:23,307 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1
>>> 2016-12-19 10:28:26,363 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1
>>> 2016-12-19 10:28:26,567 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Map 1: 0(+1)/2 Reducer 2: 0/1
>>> 2016-12-19 10:28:26,596 INFO [pool-7-thread-1]
>>> threadpool.DefaultScheduler:118 : Job Fetcher: 1 should running, 1
>>> actual running, 0 ready, 0 already succeed, 3 error, 1 discarded, 0 others
>>> 2016-12-19 10:28:26,769 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1
>>> 2016-12-19 10:28:29,810 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1
>>> 2016-12-19 10:28:30,217 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Map 1: 1(+1)/2 Reducer 2: 0(+1)/1
>>> 2016-12-19 10:28:30,826 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 0(+1)/1
>>> 2016-12-19 10:28:31,232 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 1/1
>>> 2016-12-19 10:28:31,319 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Moving data to:
>>> /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48
>>> b4-bdb5-6e578320375a/row_count
>>> 2016-12-19 10:28:31,406 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : OK
>>> 2016-12-19 10:28:31,454 INFO [pool-8-thread-7]
>>> execution.AbstractExecutable:36 : Time taken: 16.701 seconds
>>> 2016-12-19 10:28:35,074 ERROR [pool-8-thread-7]
>>> execution.AbstractExecutable:357 : job:678266c0-ba0e-48b4-bdb5-6e578320375a-01
>>> execute finished with exception
>>> java.io.FileNotFoundException: File does not exist:
>>> /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48
>>> b4-bdb5-6e578320375a/row_count/000000_0
>>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>>> deFile.java:71)
>>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>>> deFile.java:61)
>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>>> LocationsInt(FSNamesystem.java:1820)
>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>>> Locations(FSNamesystem.java:1791)
>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>>> Locations(FSNamesystem.java:1704)
>>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.get
>>> BlockLocations(NameNodeRpcServer.java:587)
>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ
>>> erSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolS
>>> erverSideTranslatorPB.java:365)
>>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol
>>> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
>>> enodeProtocolProtos.java)
>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn
>>> voker.call(ProtobufRpcEngine.java:616)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>>> upInformation.java:1657)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)
>>>
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Native
>>> ConstructorAccessorImpl.java:57)
>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(De
>>> legatingConstructorAccessorImpl.java:45)
>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>> at org.apache.hadoop.ipc.RemoteException.instantiateException(R
>>> emoteException.java:106)
>>> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(
>>> RemoteException.java:73)
>>> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSCl
>>> ient.java:1242)
>>> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.
>>> java:1227)
>>> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.
>>> java:1215)
>>> at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndG
>>> etLastBlockLength(DFSInputStream.java:303)
>>> at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStrea
>>> m.java:269)
>>> at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.
>>> java:261)
>>> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1540)
>>> at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(Distri
>>> butedFileSystem.java:303)
>>> at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(Distri
>>> butedFileSystem.java:299)
>>> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSyst
>>> emLinkResolver.java:81)
>>> at org.apache.hadoop.hdfs.DistributedFileSystem.open(Distribute
>>> dFileSystem.java:299)
>>> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
>>> at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiv
>>> eTableStep.readRowCountFromFile(HiveMRInput.java:294)
>>> at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiv
>>> eTableStep.doWork(HiveMRInput.java:337)
>>> at org.apache.kylin.job.execution.AbstractExecutable.execute(Ab
>>> stractExecutable.java:113)
>>> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWo
>>> rk(DefaultChainedExecutable.java:57)
>>> at org.apache.kylin.job.execution.AbstractExecutable.execute(Ab
>>> stractExecutable.java:113)
>>> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRun
>>> ner.run(DefaultScheduler.java:136)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1145)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException):
>>> File does not exist: /young/kylin_test/kylin_metada
>>> ta_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/000000_0
>>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>>> deFile.java:71)
>>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>>> deFile.java:61)
>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>>> LocationsInt(FSNamesystem.java:1820)
>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>>> Locations(FSNamesystem.java:1791)
>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>>> Locations(FSNamesystem.java:1704)
>>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.get
>>> BlockLocations(NameNodeRpcServer.java:587)
>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ
>>> erSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolS
>>> erverSideTranslatorPB.java:365)
>>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol
>>> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
>>> enodeProtocolProtos.java)
>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn
>>> voker.call(ProtobufRpcEngine.java:616)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>>> upInformation.java:1657)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1358)
>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(Proto
>>> bufRpcEngine.java:229)
>>> at com.sun.proxy.$Proxy25.getBlockLocations(Unknown Source)
>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran
>>> slatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:57)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth
>>> od(RetryInvocationHandler.java:252)
>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret
>>> ryInvocationHandler.java:104)
>>> at com.sun.proxy.$Proxy26.getBlockLocations(Unknown Source)
>>> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSCl
>>> ient.java:1240)
>>> ... 20 more
>>> 2016-12-19 10:28:35,078 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 :
>>> updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
>>> 2016-12-19 10:28:35,082 DEBUG [pool-8-thread-7]
>>> hbase.HBaseResourceStore:262 : Update row /execute_output/678266c0-ba0e-48b4-bdb5-6e578320375a-01
>>> from oldTs: 1482114480638, to newTs: 1482114515078, operation result: true
>>> 2016-12-19 10:28:35,085 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 :
>>> updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
>>>
>>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Error when #2 Step: Redistribute Flat Hive Table - File does not exist
Posted by Alberto Ramón <a....@gmail.com>.
other idea:
Can be a problem with permissions?: the user that execute Kylin can't read
data generated by YARN
check if Kylin user can read your folder /young/kylin_test/
Which Hadoop user are executing Kylin?
(no more ideas, Good Luck)
2016-12-20 7:51 GMT+01:00 雨日听风 <49...@qq.com>:
> Thank you!
> We checked the yarn and hard disk. But not found any error. Hard disk
> space and memory and so on is working well.
> Last time its error code was "unknownhost clusterB",now in new server env
> it cant find clusterB(hbase only). but cant find rowCount file.
> ===
> the follow command runs ok:
> hdfs dfs -mkdir /young/kylin_test/kylin_metadata_nokia/
> kylin-678c15ba-5375-4f80-831e-1ae0af8ed576/row_count/tmp
> And "ls" cant find file "000000_0" which it said "file does not exist".
>
> ------------------ 原始邮件 ------------------
> *发件人:* "Alberto Ramón";<a....@gmail.com>;
> *发送时间:* 2016年12月19日(星期一) 晚上9:13
> *收件人:* "user"<us...@kylin.apache.org>;
> *主题:* Re: Error when #2 Step: Redistribute Flat Hive Table - File does
> not exist
>
> i think i had this error last nigth :)
> (go to yarn to find detailed error & find on internet)
> in my case was free space less than 10% of hard disk. Check this please
>
> El 19/12/2016 11:35, "雨日听风" <49...@qq.com> escribió:
>
>> When I build a cube in kylin1.6, I get error in step #2: Redistribute
>> Flat Hive Table
>>
>> Please help! Thank you very much!
>>
>> env: kylin1.6 is in a independent server, and have 2 other server
>> cluster: clusterA(hive only) and clusterB(hbase only).
>> Error is:
>>
>> 2016-12-19 10:28:00,641 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Compute row count of flat hive table,
>> cmd:
>> 2016-12-19 10:28:00,642 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : hive -e "USE boco;
>> SET dfs.replication=2;
>> SET hive.exec.compress.output=true;
>> SET hive.auto.convert.join.noconditionaltask=true;
>> SET hive.auto.convert.join.noconditionaltask.size=100000000;
>> SET mapreduce.output.fileoutputformat.compress.type=BLOCK;
>> SET mapreduce.job.split.metainfo.maxsize=-1;
>> SET mapreduce.job.queuename=young;
>> SET tez.queue.name=young;
>>
>> set hive.exec.compress.output=false;
>>
>> set hive.exec.compress.output=false;
>> INSERT OVERWRITE DIRECTORY '/young/kylin_test/kylin_metad
>> ata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count' SELECT
>> count(*) FROM kylin_intermediate_hbase_in_testCluster_CUBE_f9468805_eabf_
>> 4b54_bf2b_182e4c86214a;
>>
>> "
>> 2016-12-19 10:28:03,277 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : WARNING: Use "yarn jar" to launch YARN
>> applications.
>> 2016-12-19 10:28:04,444 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 :
>> 2016-12-19 10:28:04,445 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Logging initialized using
>> configuration in file:/etc/hive/conf/hive-log4j.properties
>> 2016-12-19 10:28:14,700 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : OK
>> 2016-12-19 10:28:14,703 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Time taken: 0.935 seconds
>> 2016-12-19 10:28:15,559 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Query ID =
>> young_20161219102814_a7104fd4-ba83-47fc-ac0b-0c9bef4e1969
>> 2016-12-19 10:28:15,560 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Total jobs = 1
>> 2016-12-19 10:28:15,575 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Launching Job 1 out of 1
>> 2016-12-19 10:28:22,842 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 :
>> 2016-12-19 10:28:22,842 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 :
>> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Status: Running (Executing on YARN
>> cluster with App id application_1473415773736_1063281)
>> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 :
>> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: -/- Reducer 2: 0/1
>> 2016-12-19 10:28:23,307 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1
>> 2016-12-19 10:28:26,363 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1
>> 2016-12-19 10:28:26,567 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 0(+1)/2 Reducer 2: 0/1
>> 2016-12-19 10:28:26,596 INFO [pool-7-thread-1]
>> threadpool.DefaultScheduler:118 : Job Fetcher: 1 should running, 1
>> actual running, 0 ready, 0 already succeed, 3 error, 1 discarded, 0 others
>> 2016-12-19 10:28:26,769 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1
>> 2016-12-19 10:28:29,810 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1
>> 2016-12-19 10:28:30,217 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 1(+1)/2 Reducer 2: 0(+1)/1
>> 2016-12-19 10:28:30,826 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 0(+1)/1
>> 2016-12-19 10:28:31,232 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 1/1
>> 2016-12-19 10:28:31,319 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Moving data to:
>> /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-
>> 48b4-bdb5-6e578320375a/row_count
>> 2016-12-19 10:28:31,406 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : OK
>> 2016-12-19 10:28:31,454 INFO [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Time taken: 16.701 seconds
>> 2016-12-19 10:28:35,074 ERROR [pool-8-thread-7]
>> execution.AbstractExecutable:357 : job:678266c0-ba0e-48b4-bdb5-6e578320375a-01
>> execute finished with exception
>> java.io.FileNotFoundException: File does not exist:
>> /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-
>> 48b4-bdb5-6e578320375a/row_count/000000_0
>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>> deFile.java:71)
>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>> deFile.java:61)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>> LocationsInt(FSNamesystem.java:1820)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>> Locations(FSNamesystem.java:1791)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>> Locations(FSNamesystem.java:1704)
>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.get
>> BlockLocations(NameNodeRpcServer.java:587)
>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ
>> erSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolS
>> erverSideTranslatorPB.java:365)
>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol
>> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
>> enodeProtocolProtos.java)
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn
>> voker.call(ProtobufRpcEngine.java:616)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>> upInformation.java:1657)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)
>>
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Native
>> ConstructorAccessorImpl.java:57)
>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(De
>> legatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> at org.apache.hadoop.ipc.RemoteException.instantiateException(R
>> emoteException.java:106)
>> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(
>> RemoteException.java:73)
>> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSCl
>> ient.java:1242)
>> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.
>> java:1227)
>> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.
>> java:1215)
>> at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndG
>> etLastBlockLength(DFSInputStream.java:303)
>> at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStrea
>> m.java:269)
>> at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:261)
>> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1540)
>> at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(
>> DistributedFileSystem.java:303)
>> at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(
>> DistributedFileSystem.java:299)
>> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSyst
>> emLinkResolver.java:81)
>> at org.apache.hadoop.hdfs.DistributedFileSystem.open(Distribute
>> dFileSystem.java:299)
>> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
>> at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiv
>> eTableStep.readRowCountFromFile(HiveMRInput.java:294)
>> at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiv
>> eTableStep.doWork(HiveMRInput.java:337)
>> at org.apache.kylin.job.execution.AbstractExecutable.execute(
>> AbstractExecutable.java:113)
>> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWo
>> rk(DefaultChainedExecutable.java:57)
>> at org.apache.kylin.job.execution.AbstractExecutable.execute(
>> AbstractExecutable.java:113)
>> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRun
>> ner.run(DefaultScheduler.java:136)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException):
>> File does not exist: /young/kylin_test/kylin_metada
>> ta_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/000000_0
>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>> deFile.java:71)
>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>> deFile.java:61)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>> LocationsInt(FSNamesystem.java:1820)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>> Locations(FSNamesystem.java:1791)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>> Locations(FSNamesystem.java:1704)
>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.get
>> BlockLocations(NameNodeRpcServer.java:587)
>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ
>> erSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolS
>> erverSideTranslatorPB.java:365)
>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol
>> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
>> enodeProtocolProtos.java)
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn
>> voker.call(ProtobufRpcEngine.java:616)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>> upInformation.java:1657)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1358)
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(
>> ProtobufRpcEngine.java:229)
>> at com.sun.proxy.$Proxy25.getBlockLocations(Unknown Source)
>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran
>> slatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>> ssorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth
>> od(RetryInvocationHandler.java:252)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret
>> ryInvocationHandler.java:104)
>> at com.sun.proxy.$Proxy26.getBlockLocations(Unknown Source)
>> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSCl
>> ient.java:1240)
>> ... 20 more
>> 2016-12-19 10:28:35,078 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 :
>> updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
>> 2016-12-19 10:28:35,082 DEBUG [pool-8-thread-7]
>> hbase.HBaseResourceStore:262 : Update row /execute_output/678266c0-ba0e-48b4-bdb5-6e578320375a-01
>> from oldTs: 1482114480638, to newTs: 1482114515078, operation result: true
>> 2016-12-19 10:28:35,085 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 :
>> updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
>>
>