You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Bart <ba...@o2.pl> on 2017/03/28 22:03:59 UTC
Problem with building other cube than Sample [Cloudera]
Hi there!
I'm new in that stuff, but I'm really enjoying Kylin at all.
First of all, I'm using Cloudera, cdh 5.8.0, Hadoop 2.6.0. I have Hive,
HBase and so on - it's Cloudera based.
I've installed version of Kylin which is adapted to my Cloudera. I started
Kylin, did a Sample Cube - everything is working fine. Cube was built.
I couldn't do a tutorial cube with steps:
http://kylin.apache.org/docs20/tutorial/create_cube.html It doesn't work!
Other ones neither! It stops on Kylin_Cube_Builer_namenamename_Cube (5%)
[Hue Job Browser]
When I'm trying to do my own Cube - it stops on "Build Cube" step as well -
I can see it in Hue Job Browser. It stops every time on 5%. Even if I try to
use my own Database (which was built in Hive), the "Default" database (new
tables) and the same database and tables (sample above)
Of course I could give you another informations if you need, but please -
try to help me. I'm just finding the answer for 2 weeks and there's no end
of it :(
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Problem-with-building-other-cube-than-Sample-Cloudera-tp7533.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: Problem with building other cube than Sample [Cloudera]
Posted by Li Yang <li...@apache.org>.
I'm setting the default algorithm to 'layer' in the next release. The
current 'auto' didn't do a good job in all kinds of environments.
Thanks for reporting.
Yang
On Thu, Apr 6, 2017 at 9:10 AM, ShaoFeng Shi <sh...@apache.org> wrote:
> If the current cluster is tiny, yes it is true.
>
> 2017-04-06 4:31 GMT+08:00 Bart <ba...@o2.pl>:
>
> > First of all, thank you very much for advice.
> > I need to day that when I changed the algorithm - it worked. Sooo... I
> need
> > more memory as I think?
> >
> >
> > --
> > View this message in context: http://apache-kylin.74782.x6.
> > nabble.com/Problem-with-building-other-cube-than-
> > Sample-Cloudera-tp7533p7593.html
> > Sent from the Apache Kylin mailing list archive at Nabble.com.
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
Re: Problem with building other cube than Sample [Cloudera]
Posted by ShaoFeng Shi <sh...@apache.org>.
If the current cluster is tiny, yes it is true.
2017-04-06 4:31 GMT+08:00 Bart <ba...@o2.pl>:
> First of all, thank you very much for advice.
> I need to day that when I changed the algorithm - it worked. Sooo... I need
> more memory as I think?
>
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Problem-with-building-other-cube-than-
> Sample-Cloudera-tp7533p7593.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Problem with building other cube than Sample [Cloudera]
Posted by Bart <ba...@o2.pl>.
First of all, thank you very much for advice.
I need to day that when I changed the algorithm - it worked. Sooo... I need
more memory as I think?
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Problem-with-building-other-cube-than-Sample-Cloudera-tp7533p7593.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: Problem with building other cube than Sample [Cloudera]
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Bart,
What's the scale of your hadoop cluster? is it a small cluster with limited
cpu/memory resource?
If the sample cube can be built, it means the Kylin/Hadoop integration is
well. Your cube seems being using the "in-mem" cubing algorithm, which will
request more memory than the sample cube (which uses the "layer"
algorithm). If YARN couldn't provide the resource, it is likely stuck in
the build step. You can bypass this with modifying the parameters
(mapreduce.map.memory.mb
and mapreduce.map.java.opts to a smaller value) in
$KYLIN_HOME/conf/kylin_job_conf_inmem.xml, or chaning
"kylin.cube.algorithm" to "layer" in kylin.properties. You need restart
kylin and re-submit a build job to make the change effective.
Just take a try to see whether it works.
2017-03-31 6:29 GMT+08:00 Bart <ba...@o2.pl>:
> OK. I've done again the tutorial and i noticed one fact.
> 1) When I do it with aggregations - it works.
> 2) When i do it without aggregations (just with this one, _COUNT_ which is
> there from the beginning) - it stays at 5% while being on "building cube
> step"
>
> I tried the same with smaaaaall, small cube.
> Here are the logs from the building step.
>
>
> stderr:
>
> > Mar 30, 2017 3:14:13 PM
> > com.google.inject.servlet.InternalServletModule$
> BackwardsCompatibleServletContextProvider
> > get
> > WARNING: You are attempting to use a deprecated API (specifically,
> > attempting to @Inject ServletContext inside an eagerly created singleton.
> > While we allow this for backwards compatibility, be warned that this MAY
> > have unexpected behavior if you have more than one injector (with
> > ServletModule) running in the same JVM. Please consult the Guice
> > documentation at http://code.google.com/p/google-guice/wiki/Servlets for
> > more information.
> > Mar 30, 2017 3:14:13 PM
> > com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> register
> > INFO: Registering
> > org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a
> > provider class
> > Mar 30, 2017 3:14:13 PM
> > com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> register
> > INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler
> as
> > a provider class
> > Mar 30, 2017 3:14:13 PM
> > com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> register
> > INFO: Registering org.apache.hadoop.mapreduce.
> v2.app.webapp.AMWebServices
> > as a root resource class
> > Mar 30, 2017 3:14:13 PM
> > com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
> > INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011
> 11:17
> > AM'
> > Mar 30, 2017 3:14:13 PM
> > com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> > getComponentProvider
> > INFO: Binding
> > org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to
> > GuiceManagedComponentProvider with the scope "Singleton"
> > Mar 30, 2017 3:14:13 PM
> > com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> > getComponentProvider
> > INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to
> > GuiceManagedComponentProvider with the scope "Singleton"
> > Mar 30, 2017 3:14:14 PM
> > com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> > getComponentProvider
> > INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to
> > GuiceManagedComponentProvider with the scope "PerRequest"
>
> stdout is clear,
>
>
> and syslog:
>
> > 2017-03-30 15:14:01,701 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> > application appattempt_1490085539706_0101_000001
> > 2017-03-30 15:14:02,235 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> > 2017-03-30 15:14:02,283 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN,
> > Service: , Ident:
> > (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@2c80463c)
> > 2017-03-30 15:14:02,315 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred
> > newApiCommitter.
> > 2017-03-30 15:14:03,081 WARN [main]
> > org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop
> > library for your platform... using builtin-java classes where applicable
> > 2017-03-30 15:14:03,240 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in
> > config null
> > 2017-03-30 15:14:03,316 INFO [main]
> > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output
> > Committer Algorithm version is 1
> > 2017-03-30 15:14:03,320 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is
> > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> > 2017-03-30 15:14:03,367 INFO [main]
> > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> > org.apache.hadoop.mapreduce.jobhistory.EventType for class
> > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
> > 2017-03-30 15:14:03,369 INFO [main]
> > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> > org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
> > 2017-03-30 15:14:03,370 INFO [main]
> > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> > org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
> > 2017-03-30 15:14:03,371 INFO [main]
> > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> > org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for
> > class
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$
> TaskAttemptEventDispatcher
> > 2017-03-30 15:14:03,372 INFO [main]
> > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> > org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
> > org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
> > 2017-03-30 15:14:03,376 INFO [main]
> > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> > org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for
> > class
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
> > 2017-03-30 15:14:03,377 INFO [main]
> > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> > org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for
> > class
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
> > 2017-03-30 15:14:03,378 INFO [main]
> > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType
> > for class
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
> > 2017-03-30 15:14:03,446 INFO [main]
> > org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> > system [hdfs://quickstart.cloudera:8020]
> > 2017-03-30 15:14:03,495 INFO [main]
> > org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> > system [hdfs://quickstart.cloudera:8020]
> > 2017-03-30 15:14:03,533 INFO [main]
> > org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> > system [hdfs://quickstart.cloudera:8020]
> > 2017-03-30 15:14:03,545 INFO [main]
> > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting
> > job history data to the timeline server is not enabled
> > 2017-03-30 15:14:03,607 INFO [main]
> > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> > org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for
> class
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
> > 2017-03-30 15:14:03,892 INFO [main]
> > org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> > hadoop-metrics2.properties
> > 2017-03-30 15:14:03,968 INFO [main]
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> > period at 10 second(s).
> > 2017-03-30 15:14:03,968 INFO [main]
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics
> > system started
> > 2017-03-30 15:14:03,978 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token
> for
> > job_1490085539706_0101 to jobTokenSecretManager
> > 2017-03-30 15:14:04,144 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing
> > job_1490085539706_0101 because: not enabled; too much RAM;
> > 2017-03-30 15:14:04,171 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job
> > job_1490085539706_0101 = 217. Number of splits = 1
> > 2017-03-30 15:14:04,173 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces
> for
> > job job_1490085539706_0101 = 1
> > 2017-03-30 15:14:04,173 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> > job_1490085539706_0101Job Transitioned from NEW to INITED
> > 2017-03-30 15:14:04,175 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching
> > normal, non-uberized, multi-container job job_1490085539706_0101.
> > 2017-03-30 15:14:04,221 INFO [main]
> > org.apache.hadoop.ipc.CallQueueManager: Using callQueue class
> > java.util.concurrent.LinkedBlockingQueue
> > 2017-03-30 15:14:04,235 INFO [Socket Reader #1 for port 33073]
> > org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 33073
> > 2017-03-30 15:14:04,257 INFO [main]
> > org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
> > protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the
> > server
> > 2017-03-30 15:14:04,258 INFO [IPC Server Responder]
> > org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> > 2017-03-30 15:14:04,258 INFO [IPC Server listener on 33073]
> > org.apache.hadoop.ipc.Server: IPC Server listener on 33073: starting
> > 2017-03-30 15:14:04,259 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated
> > MRClientService at quickstart.cloudera/10.0.0.12:33073
> > 2017-03-30 15:14:04,339 INFO [main] org.mortbay.log: Logging to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 2017-03-30 15:14:04,348 INFO [main]
> > org.apache.hadoop.security.authentication.server.AuthenticationFilter:
> > Unable to initialize FileSignerSecretProvider, falling back to use random
> > secrets.
> > 2017-03-30 15:14:04,355 WARN [main] org.apache.hadoop.http.
> HttpRequestLog:
> > Jetty request log can only be enabled using Log4j
> > 2017-03-30 15:14:04,375 INFO [main] org.apache.hadoop.http.HttpServer2:
> > Added global filter 'safety'
> > (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
> > 2017-03-30 15:14:04,382 INFO [main] org.apache.hadoop.http.HttpServer2:
> > Added filter AM_PROXY_FILTER
> > (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to
> > context mapreduce
> > 2017-03-30 15:14:04,382 INFO [main] org.apache.hadoop.http.HttpServer2:
> > Added filter AM_PROXY_FILTER
> > (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to
> > context static
> > 2017-03-30 15:14:04,386 INFO [main] org.apache.hadoop.http.HttpServer2:
> > adding path spec: /mapreduce/*
> > 2017-03-30 15:14:04,386 INFO [main] org.apache.hadoop.http.HttpServer2:
> > adding path spec: /ws/*
> > 2017-03-30 15:14:04,397 INFO [main] org.apache.hadoop.http.HttpServer2:
> > Jetty bound to port 59145
> > 2017-03-30 15:14:04,398 INFO [main] org.mortbay.log:
> > jetty-6.1.26.cloudera.4
> > 2017-03-30 15:14:04,445 INFO [main] org.mortbay.log: Extract
> > jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.6.0-cdh5.
> 8.0.jar!/webapps/mapreduce
> > to /tmp/Jetty_0_0_0_0_59145_mapreduce____.28wbth/webapp
> > 2017-03-30 15:14:04,903 INFO [main] org.mortbay.log: Started
> > HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:59145
> > 2017-03-30 15:14:04,903 INFO [main] org.apache.hadoop.yarn.webapp.
> WebApps:
> > Web app /mapreduce started at 59145
> > 2017-03-30 15:14:05,336 INFO [main] org.apache.hadoop.yarn.webapp.
> WebApps:
> > Registered webapp guice modules
> > 2017-03-30 15:14:05,343 INFO [main]
> > org.apache.hadoop.ipc.CallQueueManager: Using callQueue class
> > java.util.concurrent.LinkedBlockingQueue
> > 2017-03-30 15:14:05,345 INFO [Socket Reader #1 for port 33629]
> > org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 33629
> > 2017-03-30 15:14:05,352 INFO [IPC Server Responder]
> > org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> > 2017-03-30 15:14:05,352 INFO [IPC Server listener on 33629]
> > org.apache.hadoop.ipc.Server: IPC Server listener on 33629: starting
> > 2017-03-30 15:14:05,393 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> > nodeBlacklistingEnabled:true
> > 2017-03-30 15:14:05,393 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> > maxTaskFailuresPerNode is 3
> > 2017-03-30 15:14:05,393 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> > blacklistDisablePercent is 33
> > 2017-03-30 15:14:05,445 INFO [main] org.apache.hadoop.yarn.client.
> RMProxy:
> > Connecting to ResourceManager at quickstart.cloudera/10.0.0.12:8030
> > 2017-03-30 15:14:05,535 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
> > maxContainerCapability:
> > <
> > memory:3072, vCores:2
> > >
> > 2017-03-30 15:14:05,536 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue:
> > root.users.cloudera
> > 2017-03-30 15:14:05,541 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper
> > limit on the thread pool size is 500
> > 2017-03-30 15:14:05,541 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The
> > thread pool initial size is 10
> > 2017-03-30 15:14:05,543 INFO [main]
> > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> > yarn.client.max-cached-nodemanagers-proxies : 0
> > 2017-03-30 15:14:05,550 INFO [AsyncDispatcher event handler]
> > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> > job_1490085539706_0101Job Transitioned from INITED to SETUP
> > 2017-03-30 15:14:05,559 INFO [CommitterEvent Processor #0]
> > org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler:
> > Processing the event EventType: JOB_SETUP
> > 2017-03-30 15:14:05,573 INFO [AsyncDispatcher event handler]
> > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> > job_1490085539706_0101Job Transitioned from SETUP to RUNNING
> > 2017-03-30 15:14:05,647 INFO [AsyncDispatcher event handler]
> > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> > task_1490085539706_0101_m_000000 Task Transitioned from NEW to SCHEDULED
> > 2017-03-30 15:14:05,648 INFO [AsyncDispatcher event handler]
> > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> > task_1490085539706_0101_r_000000 Task Transitioned from NEW to SCHEDULED
> > 2017-03-30 15:14:05,650 INFO [AsyncDispatcher event handler]
> > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> > attempt_1490085539706_0101_m_000000_0 TaskAttempt Transitioned from NEW
> to
> > UNASSIGNED
> > 2017-03-30 15:14:05,650 INFO [AsyncDispatcher event handler]
> > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> > attempt_1490085539706_0101_r_000000_0 TaskAttempt Transitioned from NEW
> to
> > UNASSIGNED
> > 2017-03-30 15:14:05,652 INFO [Thread-52]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
> > mapResourceRequest:
> > <
> > memory:3072, vCores:1
> > >
> > 2017-03-30 15:14:05,662 INFO [Thread-52]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
> > reduceResourceRequest:
> > <
> > memory:128, vCores:1
> > >
> > 2017-03-30 15:14:05,669 INFO [eventHandlingThread]
> > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event
> > Writer setup for JobId: job_1490085539706_0101, File:
> > hdfs://quickstart.cloudera:8020/user/cloudera/.staging/
> job_1490085539706_0101/job_1490085539706_0101_1.jhist
> > 2017-03-30 15:14:06,002 INFO [eventHandlingThread]
> > org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> > system [hdfs://quickstart.cloudera:8020]
> > 2017-03-30 15:14:06,540 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> > Scheduling: PendingReds:1 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
> > AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0
> > HostLocal:0 RackLocal:0
> > 2017-03-30 15:14:06,607 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> getResources()
> > for application_1490085539706_0101: ask=3 release= 0 newContainers=0
> > finishedContainers=0 resourcelimit=
> > <
> > memory:2560, vCores:2
> > >
> > knownNMs=1
> > 2017-03-30 15:14:06,609 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:07,615 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:08,619 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:09,624 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:10,629 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:11,632 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:12,636 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:13,640 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:14,643 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:15,647 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:16,652 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:17,656 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:18,664 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:19,668 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:20,672 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:21,680 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:22,683 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:23,688 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:24,692 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:25,695 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:26,700 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:27,704 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:28,709 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > 2017-03-30 15:14:29,714 INFO [RMCommunicator Allocator]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> > preempt 24 due to lack of space for maps
> > ...
>
> the last one is coming back and back... Is it possible that I have lack of
> space for maps? I have 2 fact in table of facts and 2 dates in my one
> dimension (table of dates, one column is the same as in fact)
>
> That's how i build a database in Hive:
>
>
> > create table if not exists fact (price int, quantity int, datedate date)
> > ROW FORMAT delimited
> > Fields terminated by ','
> > lines terminated by '\n'
> > stored as textfile
> > load data inpath '/user/cloudera/proba/fact.txt'
> > overwrite into table fact;
> >
> > create table if not exists date_date (datedate date, start_date date)
> > ROW FORMAT delimited
> > Fields terminated by ','
> > lines terminated by '\n'
> > stored as textfile;
> > load data inpath '/user/cloudera/proba/date_date.txt'
> > overwrite into table date_date;
>
> My facts are:
>
> > 2,2,2012-08-16
> > 2,2,2012-10-06
>
> My date_date:
>
> > 2012-08-16, 2012-01-01
> > 2012-10-06, 2012-01-01
>
> That's all. That's data on which I build a cube. it's probably not possible
> that i have no space for maps...
>
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Problem-with-building-other-cube-than-
> Sample-Cloudera-tp7533p7552.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Problem with building other cube than Sample [Cloudera]
Posted by Bart <ba...@o2.pl>.
OK. I've done again the tutorial and i noticed one fact.
1) When I do it with aggregations - it works.
2) When i do it without aggregations (just with this one, _COUNT_ which is
there from the beginning) - it stays at 5% while being on "building cube
step"
I tried the same with smaaaaall, small cube.
Here are the logs from the building step.
stderr:
> Mar 30, 2017 3:14:13 PM
> com.google.inject.servlet.InternalServletModule$BackwardsCompatibleServletContextProvider
> get
> WARNING: You are attempting to use a deprecated API (specifically,
> attempting to @Inject ServletContext inside an eagerly created singleton.
> While we allow this for backwards compatibility, be warned that this MAY
> have unexpected behavior if you have more than one injector (with
> ServletModule) running in the same JVM. Please consult the Guice
> documentation at http://code.google.com/p/google-guice/wiki/Servlets for
> more information.
> Mar 30, 2017 3:14:13 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering
> org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a
> provider class
> Mar 30, 2017 3:14:13 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as
> a provider class
> Mar 30, 2017 3:14:13 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices
> as a root resource class
> Mar 30, 2017 3:14:13 PM
> com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
> INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17
> AM'
> Mar 30, 2017 3:14:13 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding
> org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to
> GuiceManagedComponentProvider with the scope "Singleton"
> Mar 30, 2017 3:14:13 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to
> GuiceManagedComponentProvider with the scope "Singleton"
> Mar 30, 2017 3:14:14 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to
> GuiceManagedComponentProvider with the scope "PerRequest"
stdout is clear,
and syslog:
> 2017-03-30 15:14:01,701 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1490085539706_0101_000001
> 2017-03-30 15:14:02,235 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2017-03-30 15:14:02,283 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN,
> Service: , Ident:
> (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@2c80463c)
> 2017-03-30 15:14:02,315 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred
> newApiCommitter.
> 2017-03-30 15:14:03,081 WARN [main]
> org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2017-03-30 15:14:03,240 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in
> config null
> 2017-03-30 15:14:03,316 INFO [main]
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output
> Committer Algorithm version is 1
> 2017-03-30 15:14:03,320 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> 2017-03-30 15:14:03,367 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.jobhistory.EventType for class
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
> 2017-03-30 15:14:03,369 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
> 2017-03-30 15:14:03,370 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
> 2017-03-30 15:14:03,371 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for
> class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
> 2017-03-30 15:14:03,372 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
> 2017-03-30 15:14:03,376 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for
> class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
> 2017-03-30 15:14:03,377 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for
> class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
> 2017-03-30 15:14:03,378 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType
> for class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
> 2017-03-30 15:14:03,446 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://quickstart.cloudera:8020]
> 2017-03-30 15:14:03,495 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://quickstart.cloudera:8020]
> 2017-03-30 15:14:03,533 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://quickstart.cloudera:8020]
> 2017-03-30 15:14:03,545 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting
> job history data to the timeline server is not enabled
> 2017-03-30 15:14:03,607 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
> 2017-03-30 15:14:03,892 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2017-03-30 15:14:03,968 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2017-03-30 15:14:03,968 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics
> system started
> 2017-03-30 15:14:03,978 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for
> job_1490085539706_0101 to jobTokenSecretManager
> 2017-03-30 15:14:04,144 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing
> job_1490085539706_0101 because: not enabled; too much RAM;
> 2017-03-30 15:14:04,171 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job
> job_1490085539706_0101 = 217. Number of splits = 1
> 2017-03-30 15:14:04,173 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for
> job job_1490085539706_0101 = 1
> 2017-03-30 15:14:04,173 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1490085539706_0101Job Transitioned from NEW to INITED
> 2017-03-30 15:14:04,175 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching
> normal, non-uberized, multi-container job job_1490085539706_0101.
> 2017-03-30 15:14:04,221 INFO [main]
> org.apache.hadoop.ipc.CallQueueManager: Using callQueue class
> java.util.concurrent.LinkedBlockingQueue
> 2017-03-30 15:14:04,235 INFO [Socket Reader #1 for port 33073]
> org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 33073
> 2017-03-30 15:14:04,257 INFO [main]
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
> protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the
> server
> 2017-03-30 15:14:04,258 INFO [IPC Server Responder]
> org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2017-03-30 15:14:04,258 INFO [IPC Server listener on 33073]
> org.apache.hadoop.ipc.Server: IPC Server listener on 33073: starting
> 2017-03-30 15:14:04,259 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated
> MRClientService at quickstart.cloudera/10.0.0.12:33073
> 2017-03-30 15:14:04,339 INFO [main] org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2017-03-30 15:14:04,348 INFO [main]
> org.apache.hadoop.security.authentication.server.AuthenticationFilter:
> Unable to initialize FileSignerSecretProvider, falling back to use random
> secrets.
> 2017-03-30 15:14:04,355 WARN [main] org.apache.hadoop.http.HttpRequestLog:
> Jetty request log can only be enabled using Log4j
> 2017-03-30 15:14:04,375 INFO [main] org.apache.hadoop.http.HttpServer2:
> Added global filter 'safety'
> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
> 2017-03-30 15:14:04,382 INFO [main] org.apache.hadoop.http.HttpServer2:
> Added filter AM_PROXY_FILTER
> (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to
> context mapreduce
> 2017-03-30 15:14:04,382 INFO [main] org.apache.hadoop.http.HttpServer2:
> Added filter AM_PROXY_FILTER
> (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to
> context static
> 2017-03-30 15:14:04,386 INFO [main] org.apache.hadoop.http.HttpServer2:
> adding path spec: /mapreduce/*
> 2017-03-30 15:14:04,386 INFO [main] org.apache.hadoop.http.HttpServer2:
> adding path spec: /ws/*
> 2017-03-30 15:14:04,397 INFO [main] org.apache.hadoop.http.HttpServer2:
> Jetty bound to port 59145
> 2017-03-30 15:14:04,398 INFO [main] org.mortbay.log:
> jetty-6.1.26.cloudera.4
> 2017-03-30 15:14:04,445 INFO [main] org.mortbay.log: Extract
> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.6.0-cdh5.8.0.jar!/webapps/mapreduce
> to /tmp/Jetty_0_0_0_0_59145_mapreduce____.28wbth/webapp
> 2017-03-30 15:14:04,903 INFO [main] org.mortbay.log: Started
> HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:59145
> 2017-03-30 15:14:04,903 INFO [main] org.apache.hadoop.yarn.webapp.WebApps:
> Web app /mapreduce started at 59145
> 2017-03-30 15:14:05,336 INFO [main] org.apache.hadoop.yarn.webapp.WebApps:
> Registered webapp guice modules
> 2017-03-30 15:14:05,343 INFO [main]
> org.apache.hadoop.ipc.CallQueueManager: Using callQueue class
> java.util.concurrent.LinkedBlockingQueue
> 2017-03-30 15:14:05,345 INFO [Socket Reader #1 for port 33629]
> org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 33629
> 2017-03-30 15:14:05,352 INFO [IPC Server Responder]
> org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2017-03-30 15:14:05,352 INFO [IPC Server listener on 33629]
> org.apache.hadoop.ipc.Server: IPC Server listener on 33629: starting
> 2017-03-30 15:14:05,393 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> nodeBlacklistingEnabled:true
> 2017-03-30 15:14:05,393 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> maxTaskFailuresPerNode is 3
> 2017-03-30 15:14:05,393 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> blacklistDisablePercent is 33
> 2017-03-30 15:14:05,445 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
> Connecting to ResourceManager at quickstart.cloudera/10.0.0.12:8030
> 2017-03-30 15:14:05,535 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
> maxContainerCapability:
> <
> memory:3072, vCores:2
> >
> 2017-03-30 15:14:05,536 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue:
> root.users.cloudera
> 2017-03-30 15:14:05,541 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper
> limit on the thread pool size is 500
> 2017-03-30 15:14:05,541 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The
> thread pool initial size is 10
> 2017-03-30 15:14:05,543 INFO [main]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> yarn.client.max-cached-nodemanagers-proxies : 0
> 2017-03-30 15:14:05,550 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1490085539706_0101Job Transitioned from INITED to SETUP
> 2017-03-30 15:14:05,559 INFO [CommitterEvent Processor #0]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler:
> Processing the event EventType: JOB_SETUP
> 2017-03-30 15:14:05,573 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1490085539706_0101Job Transitioned from SETUP to RUNNING
> 2017-03-30 15:14:05,647 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1490085539706_0101_m_000000 Task Transitioned from NEW to SCHEDULED
> 2017-03-30 15:14:05,648 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1490085539706_0101_r_000000 Task Transitioned from NEW to SCHEDULED
> 2017-03-30 15:14:05,650 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1490085539706_0101_m_000000_0 TaskAttempt Transitioned from NEW to
> UNASSIGNED
> 2017-03-30 15:14:05,650 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1490085539706_0101_r_000000_0 TaskAttempt Transitioned from NEW to
> UNASSIGNED
> 2017-03-30 15:14:05,652 INFO [Thread-52]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
> mapResourceRequest:
> <
> memory:3072, vCores:1
> >
> 2017-03-30 15:14:05,662 INFO [Thread-52]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
> reduceResourceRequest:
> <
> memory:128, vCores:1
> >
> 2017-03-30 15:14:05,669 INFO [eventHandlingThread]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event
> Writer setup for JobId: job_1490085539706_0101, File:
> hdfs://quickstart.cloudera:8020/user/cloudera/.staging/job_1490085539706_0101/job_1490085539706_0101_1.jhist
> 2017-03-30 15:14:06,002 INFO [eventHandlingThread]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://quickstart.cloudera:8020]
> 2017-03-30 15:14:06,540 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:1 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
> AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0
> HostLocal:0 RackLocal:0
> 2017-03-30 15:14:06,607 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1490085539706_0101: ask=3 release= 0 newContainers=0
> finishedContainers=0 resourcelimit=
> <
> memory:2560, vCores:2
> >
> knownNMs=1
> 2017-03-30 15:14:06,609 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:07,615 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:08,619 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:09,624 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:10,629 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:11,632 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:12,636 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:13,640 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:14,643 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:15,647 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:16,652 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:17,656 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:18,664 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:19,668 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:20,672 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:21,680 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:22,683 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:23,688 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:24,692 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:25,695 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:26,700 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:27,704 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:28,709 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> 2017-03-30 15:14:29,714 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to
> preempt 24 due to lack of space for maps
> ...
the last one is coming back and back... Is it possible that I have lack of
space for maps? I have 2 fact in table of facts and 2 dates in my one
dimension (table of dates, one column is the same as in fact)
That's how i build a database in Hive:
> create table if not exists fact (price int, quantity int, datedate date)
> ROW FORMAT delimited
> Fields terminated by ','
> lines terminated by '\n'
> stored as textfile
> load data inpath '/user/cloudera/proba/fact.txt'
> overwrite into table fact;
>
> create table if not exists date_date (datedate date, start_date date)
> ROW FORMAT delimited
> Fields terminated by ','
> lines terminated by '\n'
> stored as textfile;
> load data inpath '/user/cloudera/proba/date_date.txt'
> overwrite into table date_date;
My facts are:
> 2,2,2012-08-16
> 2,2,2012-10-06
My date_date:
> 2012-08-16, 2012-01-01
> 2012-10-06, 2012-01-01
That's all. That's data on which I build a cube. it's probably not possible
that i have no space for maps...
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Problem-with-building-other-cube-than-Sample-Cloudera-tp7533p7552.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: Problem with building other cube than Sample [Cloudera]
Posted by Alberto Ramón <a....@gmail.com>.
To see actual jobs of YARN: localhost:8088
To see historical jobs (has been finished): localhost:19888
(check this ports with your Hadoop distribution and actual configuration)
2017-03-29 8:26 GMT+01:00 Bart <ba...@o2.pl>:
> To see what YARN is saying I need to kill my process. It shows me the
> problem
> with memory, but I'm not sure whether it is True or just because killing
> the
> process.
>
> I've created a database with one table of fact and one dimension (with date
> obły). I pull two "facts" inside table of fact and two dates into table of
> dimension - so it's really, really small! I did a model and a cube based on
> that model. I did it like tutorial says and it doesn't work :(
>
> I gonna do it once again and put here the YARN's log.
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Problem-with-building-other-cube-than-
> Sample-Cloudera-tp7533p7543.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
Re: Problem with building other cube than Sample [Cloudera]
Posted by Bart <ba...@o2.pl>.
To see what YARN is saying I need to kill my process. It shows me the problem
with memory, but I'm not sure whether it is True or just because killing the
process.
I've created a database with one table of fact and one dimension (with date
obły). I pull two "facts" inside table of fact and two dates into table of
dimension - so it's really, really small! I did a model and a cube based on
that model. I did it like tutorial says and it doesn't work :(
I gonna do it once again and put here the YARN's log.
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Problem-with-building-other-cube-than-Sample-Cloudera-tp7533p7543.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: Problem with building other cube than Sample [Cloudera]
Posted by Li Yang <li...@apache.org>.
Since sample cube worked so I assume the basic Hadoop, Hive, HBase, and
Kylin are all working well.
The build problem after switching to your own data and cube perhaps is
caused by the increased data size. Try build a small piece of data first
and increase the size gradually.
If the build error is within MapReduce, the root cause usually lies in the
YARN logs like Alberto pointed out.
Cheers
Yang
On Wed, Mar 29, 2017 at 6:08 AM, Alberto Ramón <a....@gmail.com>
wrote:
> Did you check the YARN's log ? :)
>
> but this doubts is not dev mailList ...
>
> 2017-03-28 23:03 GMT+01:00 Bart <ba...@o2.pl>:
>
> > Hi there!
> >
> > I'm new in that stuff, but I'm really enjoying Kylin at all.
> >
> > First of all, I'm using Cloudera, cdh 5.8.0, Hadoop 2.6.0. I have Hive,
> > HBase and so on - it's Cloudera based.
> >
> > I've installed version of Kylin which is adapted to my Cloudera. I
> started
> > Kylin, did a Sample Cube - everything is working fine. Cube was built.
> >
> > I couldn't do a tutorial cube with steps:
> > http://kylin.apache.org/docs20/tutorial/create_cube.html It doesn't
> work!
> > Other ones neither! It stops on Kylin_Cube_Builer_namenamename_Cube (5%)
> > [Hue Job Browser]
> >
> > When I'm trying to do my own Cube - it stops on "Build Cube" step as
> well -
> > I can see it in Hue Job Browser. It stops every time on 5%. Even if I try
> > to
> > use my own Database (which was built in Hive), the "Default" database
> (new
> > tables) and the same database and tables (sample above)
> >
> > Of course I could give you another informations if you need, but please -
> > try to help me. I'm just finding the answer for 2 weeks and there's no
> end
> > of it :(
> >
> > --
> > View this message in context: http://apache-kylin.74782.x6.
> > nabble.com/Problem-with-building-other-cube-than-
> > Sample-Cloudera-tp7533.html
> > Sent from the Apache Kylin mailing list archive at Nabble.com.
> >
>
Re: Problem with building other cube than Sample [Cloudera]
Posted by Alberto Ramón <a....@gmail.com>.
Did you check the YARN's log ? :)
but this doubts is not dev mailList ...
2017-03-28 23:03 GMT+01:00 Bart <ba...@o2.pl>:
> Hi there!
>
> I'm new in that stuff, but I'm really enjoying Kylin at all.
>
> First of all, I'm using Cloudera, cdh 5.8.0, Hadoop 2.6.0. I have Hive,
> HBase and so on - it's Cloudera based.
>
> I've installed version of Kylin which is adapted to my Cloudera. I started
> Kylin, did a Sample Cube - everything is working fine. Cube was built.
>
> I couldn't do a tutorial cube with steps:
> http://kylin.apache.org/docs20/tutorial/create_cube.html It doesn't work!
> Other ones neither! It stops on Kylin_Cube_Builer_namenamename_Cube (5%)
> [Hue Job Browser]
>
> When I'm trying to do my own Cube - it stops on "Build Cube" step as well -
> I can see it in Hue Job Browser. It stops every time on 5%. Even if I try
> to
> use my own Database (which was built in Hive), the "Default" database (new
> tables) and the same database and tables (sample above)
>
> Of course I could give you another informations if you need, but please -
> try to help me. I'm just finding the answer for 2 weeks and there's no end
> of it :(
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Problem-with-building-other-cube-than-
> Sample-Cloudera-tp7533.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>