You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Ruslan Dautkhanov <da...@gmail.com> on 2018/01/02 19:52:09 UTC
Cube build fails on Step 3 "Extract Fact Table Distinct Columns"
Happy New Year !
Bouncing at the users list if this is a known issue to someone:
Cube build consistently fails on Step 3 "Extract Fact Table Distinct
Columns".
First two steps complete just fine.
Mappers work fine, it only the reducers that fail - many of them with the
same error [1].
Also [2] shows stdout/stderr from one of the reducers attempts.
CDH 5.12.1.
Kylin 2.2.
Has anyone seen this issue?
[1]
Exception from container-launch.
> Container id: container_e181_1512700237856_1192_01_001227
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
> at org.apache.hadoop.util.Shell.run(Shell.java:504)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:373)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Shell output: main : command provided 1 main :
> run as user is rdautkha main : requested yarn user is rdautkha
> Writing to tmp file
> /hdfs02/yarn/nm/nmPrivate/application_1512700237856_1192/container_e181_1512700237856_1192_01_001227/container_e181_1512700237856_1192_01_001227.pid.tmp
> Container exited with a non-zero exit code 1
[2]
Log Type: stderr
> Log Upload Time: Tue Jan 02 12:34:51 -0700 2018
> Log Length: 46
> Error: Could not find or load main class 8192
Log Type: stdout
> Log Upload Time: Tue Jan 02 12:34:51 -0700 2018
> Log Length: 0
Re: Cube build fails on Step 3 "Extract Fact Table Distinct Columns"
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Ruslan, I have no idea with this error either. Were you able to build
the sample cube?
2018-01-03 14:07 GMT+08:00 Ruslan Dautkhanov <da...@gmail.com>:
> Also, here is yarn logs output from one of the failing reducer :
>
> $ yarn logs -applicationId application_1512700237856_1192 -containerId
>> container_e181_1512700237856_1192_01_001172 -nodeAddress
>> pc1udatahad08.x.y:8041
>>
>
>
>> LogType:stderr
>> Log Upload Time:Tue Jan 02 12:34:51 -0700 2018
>> LogLength:46
>> Log Contents:
>> Error: Could not find or load main class 8192
>>
>
>
>> LogType:stdout
>> Log Upload Time:Tue Jan 02 12:34:51 -0700 2018
>> LogLength:0
>> Log Contents:
>
>
> So there is nothing meaningful except that "could not find or load main
> class 8192".
> Any ideas?
>
>
>
>
> --
> Ruslan Dautkhanov
>
> On Tue, Jan 2, 2018 at 10:31 PM, Ruslan Dautkhanov <da...@gmail.com>
> wrote:
>
>> ShaoFeng, I've uploaded full log as https://github.com/Tagar/dr
>> opbox/blob/master/application_1512700237856_1192.2.log.tgz
>> Sorry it's not small log as the application was running for 45 minutes
>> and all mappers have completed before reducers started to fail.
>> None of the reducers were successful.
>> I can not spot a problem there.. any hints would be extremely helpful.
>>
>>
>>
>> --
>> Ruslan Dautkhanov
>>
>> On Tue, Jan 2, 2018 at 8:08 PM, ShaoFeng Shi <sh...@apache.org>
>> wrote:
>>
>>> There should be more logs for this job. You can try "yarn logs" command
>>> to retrieve it. As-is log doesn't have enough information.
>>>
>>> 2018-01-03 3:52 GMT+08:00 Ruslan Dautkhanov <da...@gmail.com>:
>>>
>>>> Happy New Year !
>>>>
>>>> Bouncing at the users list if this is a known issue to someone:
>>>>
>>>> Cube build consistently fails on Step 3 "Extract Fact Table Distinct
>>>> Columns".
>>>> First two steps complete just fine.
>>>> Mappers work fine, it only the reducers that fail - many of them with
>>>> the same error [1].
>>>> Also [2] shows stdout/stderr from one of the reducers attempts.
>>>>
>>>> CDH 5.12.1.
>>>> Kylin 2.2.
>>>>
>>>> Has anyone seen this issue?
>>>>
>>>>
>>>> [1]
>>>>
>>>> Exception from container-launch.
>>>>> Container id: container_e181_1512700237856_1192_01_001227
>>>>> Exit code: 1
>>>>> Stack trace: ExitCodeException exitCode=1:
>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:504)
>>>>> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Sh
>>>>> ell.java:786)
>>>>> at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExec
>>>>> utor.launchContainer(LinuxContainerExecutor.java:373)
>>>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.l
>>>>> auncher.ContainerLaunch.call(ContainerLaunch.java:302)
>>>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.l
>>>>> auncher.ContainerLaunch.call(ContainerLaunch.java:82)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>>>> Executor.java:1149)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>>>> lExecutor.java:624)
>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>> Shell output: main : command provided 1 main :
>>>>> run as user is rdautkha main : requested yarn user is rdautkha
>>>>> Writing to tmp file /hdfs02/yarn/nm/nmPrivate/appl
>>>>> ication_1512700237856_1192/container_e181_1512700237856_1192
>>>>> _01_001227/container_e181_1512700237856_1192_01_001227.pid.tmp
>>>>> Container exited with a non-zero exit code 1
>>>>
>>>>
>>>>
>>>> [2]
>>>>
>>>> Log Type: stderr
>>>>> Log Upload Time: Tue Jan 02 12:34:51 -0700 2018
>>>>> Log Length: 46
>>>>> Error: Could not find or load main class 8192
>>>>
>>>>
>>>> Log Type: stdout
>>>>> Log Upload Time: Tue Jan 02 12:34:51 -0700 2018
>>>>> Log Length: 0
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Cube build fails on Step 3 "Extract Fact Table Distinct Columns"
Posted by Ruslan Dautkhanov <da...@gmail.com>.
Also, here is yarn logs output from one of the failing reducer :
$ yarn logs -applicationId application_1512700237856_1192 -containerId
> container_e181_1512700237856_1192_01_001172 -nodeAddress
> pc1udatahad08.x.y:8041
>
> LogType:stderr
> Log Upload Time:Tue Jan 02 12:34:51 -0700 2018
> LogLength:46
> Log Contents:
> Error: Could not find or load main class 8192
>
> LogType:stdout
> Log Upload Time:Tue Jan 02 12:34:51 -0700 2018
> LogLength:0
> Log Contents:
So there is nothing meaningful except that "could not find or load main
class 8192".
Any ideas?
--
Ruslan Dautkhanov
On Tue, Jan 2, 2018 at 10:31 PM, Ruslan Dautkhanov <da...@gmail.com>
wrote:
> ShaoFeng, I've uploaded full log as https://github.com/Tagar/
> dropbox/blob/master/application_1512700237856_1192.2.log.tgz
> Sorry it's not small log as the application was running for 45 minutes and
> all mappers have completed before reducers started to fail.
> None of the reducers were successful.
> I can not spot a problem there.. any hints would be extremely helpful.
>
>
>
> --
> Ruslan Dautkhanov
>
> On Tue, Jan 2, 2018 at 8:08 PM, ShaoFeng Shi <sh...@apache.org>
> wrote:
>
>> There should be more logs for this job. You can try "yarn logs" command
>> to retrieve it. As-is log doesn't have enough information.
>>
>> 2018-01-03 3:52 GMT+08:00 Ruslan Dautkhanov <da...@gmail.com>:
>>
>>> Happy New Year !
>>>
>>> Bouncing at the users list if this is a known issue to someone:
>>>
>>> Cube build consistently fails on Step 3 "Extract Fact Table Distinct
>>> Columns".
>>> First two steps complete just fine.
>>> Mappers work fine, it only the reducers that fail - many of them with
>>> the same error [1].
>>> Also [2] shows stdout/stderr from one of the reducers attempts.
>>>
>>> CDH 5.12.1.
>>> Kylin 2.2.
>>>
>>> Has anyone seen this issue?
>>>
>>>
>>> [1]
>>>
>>> Exception from container-launch.
>>>> Container id: container_e181_1512700237856_1192_01_001227
>>>> Exit code: 1
>>>> Stack trace: ExitCodeException exitCode=1:
>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
>>>> at org.apache.hadoop.util.Shell.run(Shell.java:504)
>>>> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Sh
>>>> ell.java:786)
>>>> at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExec
>>>> utor.launchContainer(LinuxContainerExecutor.java:373)
>>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.l
>>>> auncher.ContainerLaunch.call(ContainerLaunch.java:302)
>>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.l
>>>> auncher.ContainerLaunch.call(ContainerLaunch.java:82)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>>> Executor.java:1149)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>>> lExecutor.java:624)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Shell output: main : command provided 1 main :
>>>> run as user is rdautkha main : requested yarn user is rdautkha
>>>> Writing to tmp file /hdfs02/yarn/nm/nmPrivate/appl
>>>> ication_1512700237856_1192/container_e181_1512700237856_1192
>>>> _01_001227/container_e181_1512700237856_1192_01_001227.pid.tmp
>>>> Container exited with a non-zero exit code 1
>>>
>>>
>>>
>>> [2]
>>>
>>> Log Type: stderr
>>>> Log Upload Time: Tue Jan 02 12:34:51 -0700 2018
>>>> Log Length: 46
>>>> Error: Could not find or load main class 8192
>>>
>>>
>>> Log Type: stdout
>>>> Log Upload Time: Tue Jan 02 12:34:51 -0700 2018
>>>> Log Length: 0
>>>
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>
Re: Cube build fails on Step 3 "Extract Fact Table Distinct Columns"
Posted by Ruslan Dautkhanov <da...@gmail.com>.
ShaoFeng, I've uploaded full log as
https://github.com/Tagar/dropbox/blob/master/application_1512700237856_1192.2.log.tgz
Sorry it's not small log as the application was running for 45 minutes and
all mappers have completed before reducers started to fail.
None of the reducers were successful.
I can not spot a problem there.. any hints would be extremely helpful.
--
Ruslan Dautkhanov
On Tue, Jan 2, 2018 at 8:08 PM, ShaoFeng Shi <sh...@apache.org> wrote:
> There should be more logs for this job. You can try "yarn logs" command to
> retrieve it. As-is log doesn't have enough information.
>
> 2018-01-03 3:52 GMT+08:00 Ruslan Dautkhanov <da...@gmail.com>:
>
>> Happy New Year !
>>
>> Bouncing at the users list if this is a known issue to someone:
>>
>> Cube build consistently fails on Step 3 "Extract Fact Table Distinct
>> Columns".
>> First two steps complete just fine.
>> Mappers work fine, it only the reducers that fail - many of them with the
>> same error [1].
>> Also [2] shows stdout/stderr from one of the reducers attempts.
>>
>> CDH 5.12.1.
>> Kylin 2.2.
>>
>> Has anyone seen this issue?
>>
>>
>> [1]
>>
>> Exception from container-launch.
>>> Container id: container_e181_1512700237856_1192_01_001227
>>> Exit code: 1
>>> Stack trace: ExitCodeException exitCode=1:
>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
>>> at org.apache.hadoop.util.Shell.run(Shell.java:504)
>>> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Sh
>>> ell.java:786)
>>> at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExec
>>> utor.launchContainer(LinuxContainerExecutor.java:373)
>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.l
>>> auncher.ContainerLaunch.call(ContainerLaunch.java:302)
>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.l
>>> auncher.ContainerLaunch.call(ContainerLaunch.java:82)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1149)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Shell output: main : command provided 1 main :
>>> run as user is rdautkha main : requested yarn user is rdautkha
>>> Writing to tmp file /hdfs02/yarn/nm/nmPrivate/appl
>>> ication_1512700237856_1192/container_e181_1512700237856_
>>> 1192_01_001227/container_e181_1512700237856_1192_01_001227.pid.tmp
>>> Container exited with a non-zero exit code 1
>>
>>
>>
>> [2]
>>
>> Log Type: stderr
>>> Log Upload Time: Tue Jan 02 12:34:51 -0700 2018
>>> Log Length: 46
>>> Error: Could not find or load main class 8192
>>
>>
>> Log Type: stdout
>>> Log Upload Time: Tue Jan 02 12:34:51 -0700 2018
>>> Log Length: 0
>>
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
Re: Cube build fails on Step 3 "Extract Fact Table Distinct Columns"
Posted by ShaoFeng Shi <sh...@apache.org>.
There should be more logs for this job. You can try "yarn logs" command to
retrieve it. As-is log doesn't have enough information.
2018-01-03 3:52 GMT+08:00 Ruslan Dautkhanov <da...@gmail.com>:
> Happy New Year !
>
> Bouncing at the users list if this is a known issue to someone:
>
> Cube build consistently fails on Step 3 "Extract Fact Table Distinct
> Columns".
> First two steps complete just fine.
> Mappers work fine, it only the reducers that fail - many of them with the
> same error [1].
> Also [2] shows stdout/stderr from one of the reducers attempts.
>
> CDH 5.12.1.
> Kylin 2.2.
>
> Has anyone seen this issue?
>
>
> [1]
>
> Exception from container-launch.
>> Container id: container_e181_1512700237856_1192_01_001227
>> Exit code: 1
>> Stack trace: ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
>> at org.apache.hadoop.util.Shell.run(Shell.java:504)
>> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
>> Shell.java:786)
>> at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.
>> launchContainer(LinuxContainerExecutor.java:373)
>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.
>> launcher.ContainerLaunch.call(ContainerLaunch.java:302)
>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.
>> launcher.ContainerLaunch.call(ContainerLaunch.java:82)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Shell output: main : command provided 1 main :
>> run as user is rdautkha main : requested yarn user is rdautkha
>> Writing to tmp file /hdfs02/yarn/nm/nmPrivate/application_1512700237856_
>> 1192/container_e181_1512700237856_1192_01_001227/
>> container_e181_1512700237856_1192_01_001227.pid.tmp
>> Container exited with a non-zero exit code 1
>
>
>
> [2]
>
> Log Type: stderr
>> Log Upload Time: Tue Jan 02 12:34:51 -0700 2018
>> Log Length: 46
>> Error: Could not find or load main class 8192
>
>
> Log Type: stdout
>> Log Upload Time: Tue Jan 02 12:34:51 -0700 2018
>> Log Length: 0
>
>
>
--
Best regards,
Shaofeng Shi 史少锋