You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@myriad.apache.org by Mayank Bansal <ma...@uber.com> on 2016/02/22 19:33:52 UTC
Not able to launch Node Managers via Myriad
Hi,
I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x
I have followed all the instructions in the wiki
So the resource manager is up and i am able to launch the web UI of myriad
as well however whenever I try to use flex up it launches some tasks and
put them in to pending queue and after that nothing launches.
I am attaching below the resourcemanager.out and log snippets.
Please let me know if I am missing something.
Thanks,
*I see some exceptions in resourcemanager.out files those are as follows*
Feb 22, 2016 6:21:42 PM
com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
resolve
SEVERE: null
java.lang.IllegalAccessException: Class
com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
can not access a member of class javax.ws.rs.core.Response with modifiers
"protected"
at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
at java.lang.Class.newInstance(Class.java:436)
at
com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
at
com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
at
com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
at
com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
at
com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
at
com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
at
com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Feb 22, 2016 6:21:42 PM
com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
attachTypes
INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
Feb 22, 2016 6:21:42 PM
com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
attachTypes
INFO: Couldn't find JAX-B element for class
org.apache.myriad.configuration.MyriadConfiguration
Feb 22, 2016 6:21:42 PM
com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
attachTypes
INFO: Couldn't find JAX-B element for class
org.apache.myriad.api.model.GetSchedulerStateResponse
Feb 22, 2016 6:21:42 PM
com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
attachTypes
INFO: Couldn't find JAX-B element for class
org.apache.myriad.api.model.FlexUpServiceRequest
Feb 22, 2016 6:21:42 PM
com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
resolve
SEVERE: null
E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on fd=231:
Transport endpoint is not connected [107]
E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on fd=231:
Transport endpoint is not connected [107]
E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on fd=231:
Transport endpoint is not connected [107]
*This is the resource manager logs*
2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8033: starting
2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource:
Received flexup request. Profile: medium, Instances: 1, Constraints: null
2016-02-22 17:39:31,266 INFO org.apache.myriad.scheduler.MyriadOperations:
Adding 1 NM instances to cluster
2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState: Marked
taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size of
pending queue for nm is: 1
2016-02-22 17:47:25,450 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
Release request cache is cleaned up
2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource:
Received flexup request. Profile: medium, Instances: 1, Constraints: null
2016-02-22 18:22:22,094 INFO org.apache.myriad.scheduler.MyriadOperations:
Adding 1 NM instances to cluster
2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState: Marked
taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size of
pending queue for nm is: 2
Thanks,
--
- Mayank
Re: Not able to launch Node Managers via Myriad
Posted by Santosh Marella <sm...@maprtech.com>.
This probably requires a code change in Myriad.
Currently Myriad uses "$YARN_HOME/bin/yarn nodemanager" as the command to
launch the NM. This makes the NM's log4j output to be streamed to stdout. (
https://github.com/apache/incubator-myriad/blob/master/myriad-scheduler/src/main/java/org/apache/myriad/scheduler/NMExecutorCLGenImpl.java#L65
)
I think we should use "$YARN_HOME/sbin/yarn-daemon.sh start nodemanager" so
that the log4j output is written to $YARN_HOME/logs/xxx-nodemanger-yyy.log.
Do you happen to have a Myriad git clone to try it out and see if it helps?
Thanks,
Santosh
On Tue, Feb 23, 2016 at 1:49 PM, Mayank Bansal <ma...@uber.com> wrote:
> Thanks Santosh for the prompt reply and all the great help.
>
> Is there a way I can redirect mesos task logs to NodeManager logs.
> Users/Admins/tools are used to look at the logs in specific location. Its
> very hard to change all of it. Is there any way we can achieve that?
>
> Thanks,
> Mayank
>
> On Tue, Feb 23, 2016 at 11:21 AM, Santosh Marella <sm...@maprtech.com>
> wrote:
>
>> Great to hear your cluster is up.
>>
>> 1. I am not seeing Nodemanager logs, I see YARN LOG DIR is set however
>> logs are not going there
>> NM logs are accessible via the Mesos UI (click on the sandbox for
>> nm.medium.xxx task and then on stderr link).
>> Since NMs are launched via Mesos, the NM logs currently do not go to
>> YARN_HOME/logs dir.
>>
>> 2. Log aggregation is also not working in YARN , don't know why , as I
>> don't have logs
>> This should work, as Myriad is not really touching any of the traditional
>> YARN behavior/features.
>>
>> 3. Other issue is when i do first time flex up then it launches 2 node
>> managers instead of one.
>> This is expected. Once Myriad/RM starts, one medium profile NM is
>> launched automatically without requiring an explicit flexup.
>>
>> Thanks,
>> Santosh
>>
>> On Tue, Feb 23, 2016 at 11:06 AM, Mayank Bansal <ma...@uber.com>
>> wrote:
>>
>>> Hi Santosh,
>>>
>>> Thanks for looking into it.
>>>
>>> I got it working and now I am able to run Node Managers through Myriad.
>>>
>>> There are couple of issues which i am still facing , I am sure you guys
>>> would have faced those and already have the solutions for those.
>>>
>>> 1. I am not seeing Nodemanager logs, I see YARN LOG DIR is set however
>>> logs are not going there
>>> 2. Log aggregation is also not working in YARN , don't know why , as I
>>> don't have logs
>>> 3. Other issue is when i do first time flex up then it launches 2 node
>>> managers instead of one.
>>>
>>> Please let me know if u know about those issues.
>>>
>>> Thanks,
>>> Mayank
>>>
>>>
>>> On Mon, Feb 22, 2016 at 4:21 PM, Santosh Marella <sm...@maprtech.com>
>>> wrote:
>>>
>>>> Hey guys.. the flow of things is as below:
>>>>
>>>> 1. Myriad registers itself as a Mesos framework. (Frameworks view on
>>>> Mesos
>>>> UI should show this)
>>>> 2. Myriad by default attempts to launch 1 medium profile NM as soon as
>>>> RM
>>>> process is up.
>>>> This is queued up until Myriad receives an offer from Mesos that is
>>>> big
>>>> enough to launch a medium profle NM.
>>>> (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad
>>>> attempted to launch a NM).
>>>>
>>>> Mayank - in your cluster, does Mesos UI show Myriad as a registered
>>>> framework? Does it show if Myriad launched NM tasks? Perhaps a good
>>>> starting point is to check what are the sizes of the offers Myriad is
>>>> receiving (Mesos' master log should show this). You can also enable
>>>> DEBUG
>>>> logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's
>>>> log4j.properties.
>>>>
>>>>
>>>>
>>>> Santosh
>>>>
>>>> On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier <
>>>> b.hagemeier@fz-juelich.de
>>>> > wrote:
>>>>
>>>> > Hi Mayank,
>>>> >
>>>> > it seems you and I are having similar issues. Let's see whether they
>>>> can
>>>> > be traced to the same causes.
>>>> >
>>>> >
>>>> > Best regards,
>>>> > Björn
>>>> >
>>>> > Am 22.02.2016 um 19:33 schrieb Mayank Bansal:
>>>> > > Hi,
>>>> > >
>>>> > > I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x
>>>> > >
>>>> > >
>>>> > > I have followed all the instructions in the wiki
>>>> > >
>>>> > > So the resource manager is up and i am able to launch the web UI of
>>>> > myriad
>>>> > > as well however whenever I try to use flex up it launches some
>>>> tasks and
>>>> > > put them in to pending queue and after that nothing launches.
>>>> > >
>>>> > > I am attaching below the resourcemanager.out and log snippets.
>>>> > >
>>>> > > Please let me know if I am missing something.
>>>> > >
>>>> > > Thanks,
>>>> > >
>>>> > >
>>>> > > *I see some exceptions in resourcemanager.out files those are as
>>>> follows*
>>>> > >
>>>> > >
>>>> > > Feb 22, 2016 6:21:42 PM
>>>> > >
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>> > > resolve
>>>> > >
>>>> > > SEVERE: null
>>>> > >
>>>> > > java.lang.IllegalAccessException: Class
>>>> > >
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>> > > can not access a member of class javax.ws.rs.core.Response with
>>>> modifiers
>>>> > > "protected"
>>>> > >
>>>> > > at
>>>> sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
>>>> > >
>>>> > > at java.lang.Class.newInstance(Class.java:436)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
>>>> > >
>>>> > > at
>>>> > >
>>>> com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89)
>>>> > >
>>>> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> > >
>>>> > > at java.lang.reflect.Method.invoke(Method.java:498)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>>>> > >
>>>> > > at
>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>>>> > >
>>>> > > at
>>>> > > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>> > >
>>>> > > at
>>>> > >
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>> > >
>>>> > > at
>>>> > >
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>> > >
>>>> > > at
>>>> > >
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>> > >
>>>> > > at org.mortbay.jetty.Server.handle(Server.java:326)
>>>> > >
>>>> > > at
>>>> > >
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>> > >
>>>> > > at
>>>> org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>> > >
>>>> > > at
>>>> > org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>> > >
>>>> > > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>> > >
>>>> > > at
>>>> > >
>>>> >
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> > >
>>>> > >
>>>> > > Feb 22, 2016 6:21:42 PM
>>>> > >
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> > > attachTypes
>>>> > >
>>>> > > INFO: Couldn't find JAX-B element for class
>>>> javax.ws.rs.core.Response
>>>> > >
>>>> > > Feb 22, 2016 6:21:42 PM
>>>> > >
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> > > attachTypes
>>>> > >
>>>> > > INFO: Couldn't find JAX-B element for class
>>>> > > org.apache.myriad.configuration.MyriadConfiguration
>>>> > >
>>>> > > Feb 22, 2016 6:21:42 PM
>>>> > >
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> > > attachTypes
>>>> > >
>>>> > > INFO: Couldn't find JAX-B element for class
>>>> > > org.apache.myriad.api.model.GetSchedulerStateResponse
>>>> > >
>>>> > > Feb 22, 2016 6:21:42 PM
>>>> > >
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> > > attachTypes
>>>> > >
>>>> > > INFO: Couldn't find JAX-B element for class
>>>> > > org.apache.myriad.api.model.FlexUpServiceRequest
>>>> > >
>>>> > > Feb 22, 2016 6:21:42 PM
>>>> > >
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>> > > resolve
>>>> > >
>>>> > > SEVERE: null
>>>> > >
>>>> > > E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on
>>>> fd=231:
>>>> > > Transport endpoint is not connected [107]
>>>> > >
>>>> > > E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on
>>>> fd=231:
>>>> > > Transport endpoint is not connected [107]
>>>> > >
>>>> > > E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on
>>>> fd=231:
>>>> > > Transport endpoint is not connected [107]
>>>> > >
>>>> > > *This is the resource manager logs*
>>>> > >
>>>> > > 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC
>>>> Server
>>>> > > listener on 8033: starting
>>>> > >
>>>> > > 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource:
>>>> > > Received flexup request. Profile: medium, Instances: 1,
>>>> Constraints: null
>>>> > >
>>>> > > 2016-02-22 17:39:31,266 INFO
>>>> > org.apache.myriad.scheduler.MyriadOperations:
>>>> > > Adding 1 NM instances to cluster
>>>> > >
>>>> > > 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState:
>>>> > Marked
>>>> > > taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size
>>>> of
>>>> > > pending queue for nm is: 1
>>>> > >
>>>> > > 2016-02-22 17:47:25,450 INFO
>>>> > >
>>>> >
>>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
>>>> > > Release request cache is cleaned up
>>>> > >
>>>> > > 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource:
>>>> > > Received flexup request. Profile: medium, Instances: 1,
>>>> Constraints: null
>>>> > >
>>>> > > 2016-02-22 18:22:22,094 INFO
>>>> > org.apache.myriad.scheduler.MyriadOperations:
>>>> > > Adding 1 NM instances to cluster
>>>> > >
>>>> > > 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState:
>>>> > Marked
>>>> > > taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size
>>>> of
>>>> > > pending queue for nm is: 2
>>>> > >
>>>> > >
>>>> > >
>>>> > > Thanks,
>>>> > >
>>>> >
>>>> >
>>>> > --
>>>> > Dipl.-Inform. Björn Hagemeier
>>>> > Federated Systems and Data
>>>> > Juelich Supercomputing Centre
>>>> > Institute for Advanced Simulation
>>>> >
>>>> > Phone: +49 2461 61 1584
>>>> > Fax : +49 2461 61 6656
>>>> > Email: b.hagemeier@fz-juelich.de
>>>> > Skype: bhagemeier
>>>> > WWW : http://www.fz-juelich.de/jsc
>>>> >
>>>> > JSC is the coordinator of the
>>>> > John von Neumann Institute for Computing
>>>> > and member of the
>>>> > Gauss Centre for Supercomputing
>>>> >
>>>> >
>>>> >
>>>> -------------------------------------------------------------------------------------
>>>> >
>>>> >
>>>> -------------------------------------------------------------------------------------
>>>> > Forschungszentrum Juelich GmbH
>>>> > 52425 Juelich
>>>> > Sitz der Gesellschaft: Juelich
>>>> > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>>>> > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>>>> > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>>>> > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>>>> > Prof. Dr. Sebastian M. Schmidt
>>>> >
>>>> >
>>>> -------------------------------------------------------------------------------------
>>>> >
>>>> >
>>>> -------------------------------------------------------------------------------------
>>>> >
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> - Mayank
>>>
>>
>>
>
>
> --
> - Mayank
>
Re: Not able to launch Node Managers via Myriad
Posted by Mayank Bansal <ma...@uber.com>.
Thanks Santosh for the prompt reply and all the great help.
Is there a way I can redirect mesos task logs to NodeManager logs.
Users/Admins/tools are used to look at the logs in specific location. Its
very hard to change all of it. Is there any way we can achieve that?
Thanks,
Mayank
On Tue, Feb 23, 2016 at 11:21 AM, Santosh Marella <sm...@maprtech.com>
wrote:
> Great to hear your cluster is up.
>
> 1. I am not seeing Nodemanager logs, I see YARN LOG DIR is set however
> logs are not going there
> NM logs are accessible via the Mesos UI (click on the sandbox for
> nm.medium.xxx task and then on stderr link).
> Since NMs are launched via Mesos, the NM logs currently do not go to
> YARN_HOME/logs dir.
>
> 2. Log aggregation is also not working in YARN , don't know why , as I
> don't have logs
> This should work, as Myriad is not really touching any of the traditional
> YARN behavior/features.
>
> 3. Other issue is when i do first time flex up then it launches 2 node
> managers instead of one.
> This is expected. Once Myriad/RM starts, one medium profile NM is launched
> automatically without requiring an explicit flexup.
>
> Thanks,
> Santosh
>
> On Tue, Feb 23, 2016 at 11:06 AM, Mayank Bansal <ma...@uber.com> wrote:
>
>> Hi Santosh,
>>
>> Thanks for looking into it.
>>
>> I got it working and now I am able to run Node Managers through Myriad.
>>
>> There are couple of issues which i am still facing , I am sure you guys
>> would have faced those and already have the solutions for those.
>>
>> 1. I am not seeing Nodemanager logs, I see YARN LOG DIR is set however
>> logs are not going there
>> 2. Log aggregation is also not working in YARN , don't know why , as I
>> don't have logs
>> 3. Other issue is when i do first time flex up then it launches 2 node
>> managers instead of one.
>>
>> Please let me know if u know about those issues.
>>
>> Thanks,
>> Mayank
>>
>>
>> On Mon, Feb 22, 2016 at 4:21 PM, Santosh Marella <sm...@maprtech.com>
>> wrote:
>>
>>> Hey guys.. the flow of things is as below:
>>>
>>> 1. Myriad registers itself as a Mesos framework. (Frameworks view on
>>> Mesos
>>> UI should show this)
>>> 2. Myriad by default attempts to launch 1 medium profile NM as soon as RM
>>> process is up.
>>> This is queued up until Myriad receives an offer from Mesos that is
>>> big
>>> enough to launch a medium profle NM.
>>> (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad
>>> attempted to launch a NM).
>>>
>>> Mayank - in your cluster, does Mesos UI show Myriad as a registered
>>> framework? Does it show if Myriad launched NM tasks? Perhaps a good
>>> starting point is to check what are the sizes of the offers Myriad is
>>> receiving (Mesos' master log should show this). You can also enable DEBUG
>>> logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's
>>> log4j.properties.
>>>
>>>
>>>
>>> Santosh
>>>
>>> On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier <
>>> b.hagemeier@fz-juelich.de
>>> > wrote:
>>>
>>> > Hi Mayank,
>>> >
>>> > it seems you and I are having similar issues. Let's see whether they
>>> can
>>> > be traced to the same causes.
>>> >
>>> >
>>> > Best regards,
>>> > Björn
>>> >
>>> > Am 22.02.2016 um 19:33 schrieb Mayank Bansal:
>>> > > Hi,
>>> > >
>>> > > I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x
>>> > >
>>> > >
>>> > > I have followed all the instructions in the wiki
>>> > >
>>> > > So the resource manager is up and i am able to launch the web UI of
>>> > myriad
>>> > > as well however whenever I try to use flex up it launches some tasks
>>> and
>>> > > put them in to pending queue and after that nothing launches.
>>> > >
>>> > > I am attaching below the resourcemanager.out and log snippets.
>>> > >
>>> > > Please let me know if I am missing something.
>>> > >
>>> > > Thanks,
>>> > >
>>> > >
>>> > > *I see some exceptions in resourcemanager.out files those are as
>>> follows*
>>> > >
>>> > >
>>> > > Feb 22, 2016 6:21:42 PM
>>> > >
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>> > > resolve
>>> > >
>>> > > SEVERE: null
>>> > >
>>> > > java.lang.IllegalAccessException: Class
>>> > >
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>> > > can not access a member of class javax.ws.rs.core.Response with
>>> modifiers
>>> > > "protected"
>>> > >
>>> > > at
>>> sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
>>> > >
>>> > > at java.lang.Class.newInstance(Class.java:436)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
>>> > >
>>> > > at
>>> > > com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89)
>>> > >
>>> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> > >
>>> > > at
>>> > >
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> > >
>>> > > at
>>> > >
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > >
>>> > > at java.lang.reflect.Method.invoke(Method.java:498)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>>> > >
>>> > > at
>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>>> > >
>>> > > at
>>> > >
>>> >
>>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>>> > >
>>> > > at
>>> > > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>>> > >
>>> > > at
>>> > >
>>> >
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>> > >
>>> > > at
>>> > >
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>> > >
>>> > > at
>>> > >
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>> > >
>>> > > at
>>> > >
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>> > >
>>> > > at org.mortbay.jetty.Server.handle(Server.java:326)
>>> > >
>>> > > at
>>> > >
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>> > >
>>> > > at
>>> > >
>>> >
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>> > >
>>> > > at
>>> org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>> > >
>>> > > at
>>> > org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>> > >
>>> > > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>> > >
>>> > > at
>>> > >
>>> >
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>> > >
>>> > > at
>>> > >
>>> >
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> > >
>>> > >
>>> > > Feb 22, 2016 6:21:42 PM
>>> > >
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>> > > attachTypes
>>> > >
>>> > > INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
>>> > >
>>> > > Feb 22, 2016 6:21:42 PM
>>> > >
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>> > > attachTypes
>>> > >
>>> > > INFO: Couldn't find JAX-B element for class
>>> > > org.apache.myriad.configuration.MyriadConfiguration
>>> > >
>>> > > Feb 22, 2016 6:21:42 PM
>>> > >
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>> > > attachTypes
>>> > >
>>> > > INFO: Couldn't find JAX-B element for class
>>> > > org.apache.myriad.api.model.GetSchedulerStateResponse
>>> > >
>>> > > Feb 22, 2016 6:21:42 PM
>>> > >
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>> > > attachTypes
>>> > >
>>> > > INFO: Couldn't find JAX-B element for class
>>> > > org.apache.myriad.api.model.FlexUpServiceRequest
>>> > >
>>> > > Feb 22, 2016 6:21:42 PM
>>> > >
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>> > > resolve
>>> > >
>>> > > SEVERE: null
>>> > >
>>> > > E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on
>>> fd=231:
>>> > > Transport endpoint is not connected [107]
>>> > >
>>> > > E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on
>>> fd=231:
>>> > > Transport endpoint is not connected [107]
>>> > >
>>> > > E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on
>>> fd=231:
>>> > > Transport endpoint is not connected [107]
>>> > >
>>> > > *This is the resource manager logs*
>>> > >
>>> > > 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> > > listener on 8033: starting
>>> > >
>>> > > 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource:
>>> > > Received flexup request. Profile: medium, Instances: 1, Constraints:
>>> null
>>> > >
>>> > > 2016-02-22 17:39:31,266 INFO
>>> > org.apache.myriad.scheduler.MyriadOperations:
>>> > > Adding 1 NM instances to cluster
>>> > >
>>> > > 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState:
>>> > Marked
>>> > > taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size
>>> of
>>> > > pending queue for nm is: 1
>>> > >
>>> > > 2016-02-22 17:47:25,450 INFO
>>> > >
>>> >
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
>>> > > Release request cache is cleaned up
>>> > >
>>> > > 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource:
>>> > > Received flexup request. Profile: medium, Instances: 1, Constraints:
>>> null
>>> > >
>>> > > 2016-02-22 18:22:22,094 INFO
>>> > org.apache.myriad.scheduler.MyriadOperations:
>>> > > Adding 1 NM instances to cluster
>>> > >
>>> > > 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState:
>>> > Marked
>>> > > taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size
>>> of
>>> > > pending queue for nm is: 2
>>> > >
>>> > >
>>> > >
>>> > > Thanks,
>>> > >
>>> >
>>> >
>>> > --
>>> > Dipl.-Inform. Björn Hagemeier
>>> > Federated Systems and Data
>>> > Juelich Supercomputing Centre
>>> > Institute for Advanced Simulation
>>> >
>>> > Phone: +49 2461 61 1584
>>> > Fax : +49 2461 61 6656
>>> > Email: b.hagemeier@fz-juelich.de
>>> > Skype: bhagemeier
>>> > WWW : http://www.fz-juelich.de/jsc
>>> >
>>> > JSC is the coordinator of the
>>> > John von Neumann Institute for Computing
>>> > and member of the
>>> > Gauss Centre for Supercomputing
>>> >
>>> >
>>> >
>>> -------------------------------------------------------------------------------------
>>> >
>>> >
>>> -------------------------------------------------------------------------------------
>>> > Forschungszentrum Juelich GmbH
>>> > 52425 Juelich
>>> > Sitz der Gesellschaft: Juelich
>>> > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>>> > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>>> > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>>> > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>>> > Prof. Dr. Sebastian M. Schmidt
>>> >
>>> >
>>> -------------------------------------------------------------------------------------
>>> >
>>> >
>>> -------------------------------------------------------------------------------------
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> - Mayank
>>
>
>
--
- Mayank
Re: Not able to launch Node Managers via Myriad
Posted by Santosh Marella <sm...@maprtech.com>.
Great to hear your cluster is up.
1. I am not seeing Nodemanager logs, I see YARN LOG DIR is set however logs
are not going there
NM logs are accessible via the Mesos UI (click on the sandbox for
nm.medium.xxx task and then on stderr link).
Since NMs are launched via Mesos, the NM logs currently do not go to
YARN_HOME/logs dir.
2. Log aggregation is also not working in YARN , don't know why , as I
don't have logs
This should work, as Myriad is not really touching any of the traditional
YARN behavior/features.
3. Other issue is when i do first time flex up then it launches 2 node
managers instead of one.
This is expected. Once Myriad/RM starts, one medium profile NM is launched
automatically without requiring an explicit flexup.
Thanks,
Santosh
On Tue, Feb 23, 2016 at 11:06 AM, Mayank Bansal <ma...@uber.com> wrote:
> Hi Santosh,
>
> Thanks for looking into it.
>
> I got it working and now I am able to run Node Managers through Myriad.
>
> There are couple of issues which i am still facing , I am sure you guys
> would have faced those and already have the solutions for those.
>
> 1. I am not seeing Nodemanager logs, I see YARN LOG DIR is set however
> logs are not going there
> 2. Log aggregation is also not working in YARN , don't know why , as I
> don't have logs
> 3. Other issue is when i do first time flex up then it launches 2 node
> managers instead of one.
>
> Please let me know if u know about those issues.
>
> Thanks,
> Mayank
>
>
> On Mon, Feb 22, 2016 at 4:21 PM, Santosh Marella <sm...@maprtech.com>
> wrote:
>
>> Hey guys.. the flow of things is as below:
>>
>> 1. Myriad registers itself as a Mesos framework. (Frameworks view on Mesos
>> UI should show this)
>> 2. Myriad by default attempts to launch 1 medium profile NM as soon as RM
>> process is up.
>> This is queued up until Myriad receives an offer from Mesos that is
>> big
>> enough to launch a medium profle NM.
>> (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad
>> attempted to launch a NM).
>>
>> Mayank - in your cluster, does Mesos UI show Myriad as a registered
>> framework? Does it show if Myriad launched NM tasks? Perhaps a good
>> starting point is to check what are the sizes of the offers Myriad is
>> receiving (Mesos' master log should show this). You can also enable DEBUG
>> logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's
>> log4j.properties.
>>
>>
>>
>> Santosh
>>
>> On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier <
>> b.hagemeier@fz-juelich.de
>> > wrote:
>>
>> > Hi Mayank,
>> >
>> > it seems you and I are having similar issues. Let's see whether they can
>> > be traced to the same causes.
>> >
>> >
>> > Best regards,
>> > Björn
>> >
>> > Am 22.02.2016 um 19:33 schrieb Mayank Bansal:
>> > > Hi,
>> > >
>> > > I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x
>> > >
>> > >
>> > > I have followed all the instructions in the wiki
>> > >
>> > > So the resource manager is up and i am able to launch the web UI of
>> > myriad
>> > > as well however whenever I try to use flex up it launches some tasks
>> and
>> > > put them in to pending queue and after that nothing launches.
>> > >
>> > > I am attaching below the resourcemanager.out and log snippets.
>> > >
>> > > Please let me know if I am missing something.
>> > >
>> > > Thanks,
>> > >
>> > >
>> > > *I see some exceptions in resourcemanager.out files those are as
>> follows*
>> > >
>> > >
>> > > Feb 22, 2016 6:21:42 PM
>> > >
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>> > > resolve
>> > >
>> > > SEVERE: null
>> > >
>> > > java.lang.IllegalAccessException: Class
>> > >
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>> > > can not access a member of class javax.ws.rs.core.Response with
>> modifiers
>> > > "protected"
>> > >
>> > > at
>> sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
>> > >
>> > > at java.lang.Class.newInstance(Class.java:436)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
>> > >
>> > > at
>> > > com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89)
>> > >
>> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > >
>> > > at
>> > >
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> > >
>> > > at
>> > >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > >
>> > > at java.lang.reflect.Method.invoke(Method.java:498)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>> > >
>> > > at
>> > >
>> >
>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>> > >
>> > > at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> > >
>> > > at
>> > >
>> >
>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>> > >
>> > > at
>> > >
>> >
>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>> > >
>> > > at
>> > >
>> >
>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>> > >
>> > > at
>> > >
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>> > >
>> > > at
>> > >
>> >
>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>> > >
>> > > at
>> > > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>> > >
>> > > at
>> > >
>> >
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>> > >
>> > > at
>> > >
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>> > >
>> > > at
>> > >
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> > >
>> > > at
>> > >
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> > >
>> > > at org.mortbay.jetty.Server.handle(Server.java:326)
>> > >
>> > > at
>> > >
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>> > >
>> > > at
>> > >
>> >
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>> > >
>> > > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>> > >
>> > > at
>> > org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>> > >
>> > > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>> > >
>> > > at
>> > >
>> >
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>> > >
>> > > at
>> > >
>> >
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> > >
>> > >
>> > > Feb 22, 2016 6:21:42 PM
>> > >
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>> > > attachTypes
>> > >
>> > > INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
>> > >
>> > > Feb 22, 2016 6:21:42 PM
>> > >
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>> > > attachTypes
>> > >
>> > > INFO: Couldn't find JAX-B element for class
>> > > org.apache.myriad.configuration.MyriadConfiguration
>> > >
>> > > Feb 22, 2016 6:21:42 PM
>> > >
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>> > > attachTypes
>> > >
>> > > INFO: Couldn't find JAX-B element for class
>> > > org.apache.myriad.api.model.GetSchedulerStateResponse
>> > >
>> > > Feb 22, 2016 6:21:42 PM
>> > >
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>> > > attachTypes
>> > >
>> > > INFO: Couldn't find JAX-B element for class
>> > > org.apache.myriad.api.model.FlexUpServiceRequest
>> > >
>> > > Feb 22, 2016 6:21:42 PM
>> > >
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>> > > resolve
>> > >
>> > > SEVERE: null
>> > >
>> > > E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on fd=231:
>> > > Transport endpoint is not connected [107]
>> > >
>> > > E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on fd=231:
>> > > Transport endpoint is not connected [107]
>> > >
>> > > E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on fd=231:
>> > > Transport endpoint is not connected [107]
>> > >
>> > > *This is the resource manager logs*
>> > >
>> > > 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > > listener on 8033: starting
>> > >
>> > > 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource:
>> > > Received flexup request. Profile: medium, Instances: 1, Constraints:
>> null
>> > >
>> > > 2016-02-22 17:39:31,266 INFO
>> > org.apache.myriad.scheduler.MyriadOperations:
>> > > Adding 1 NM instances to cluster
>> > >
>> > > 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState:
>> > Marked
>> > > taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size of
>> > > pending queue for nm is: 1
>> > >
>> > > 2016-02-22 17:47:25,450 INFO
>> > >
>> >
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
>> > > Release request cache is cleaned up
>> > >
>> > > 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource:
>> > > Received flexup request. Profile: medium, Instances: 1, Constraints:
>> null
>> > >
>> > > 2016-02-22 18:22:22,094 INFO
>> > org.apache.myriad.scheduler.MyriadOperations:
>> > > Adding 1 NM instances to cluster
>> > >
>> > > 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState:
>> > Marked
>> > > taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size of
>> > > pending queue for nm is: 2
>> > >
>> > >
>> > >
>> > > Thanks,
>> > >
>> >
>> >
>> > --
>> > Dipl.-Inform. Björn Hagemeier
>> > Federated Systems and Data
>> > Juelich Supercomputing Centre
>> > Institute for Advanced Simulation
>> >
>> > Phone: +49 2461 61 1584
>> > Fax : +49 2461 61 6656
>> > Email: b.hagemeier@fz-juelich.de
>> > Skype: bhagemeier
>> > WWW : http://www.fz-juelich.de/jsc
>> >
>> > JSC is the coordinator of the
>> > John von Neumann Institute for Computing
>> > and member of the
>> > Gauss Centre for Supercomputing
>> >
>> >
>> >
>> -------------------------------------------------------------------------------------
>> >
>> >
>> -------------------------------------------------------------------------------------
>> > Forschungszentrum Juelich GmbH
>> > 52425 Juelich
>> > Sitz der Gesellschaft: Juelich
>> > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>> > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>> > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>> > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>> > Prof. Dr. Sebastian M. Schmidt
>> >
>> >
>> -------------------------------------------------------------------------------------
>> >
>> >
>> -------------------------------------------------------------------------------------
>> >
>> >
>>
>
>
>
> --
> - Mayank
>
Re: Not able to launch Node Managers via Myriad
Posted by Mayank Bansal <ma...@uber.com>.
Hi Santosh,
Thanks for looking into it.
I got it working and now I am able to run Node Managers through Myriad.
There are couple of issues which i am still facing , I am sure you guys
would have faced those and already have the solutions for those.
1. I am not seeing Nodemanager logs, I see YARN LOG DIR is set however logs
are not going there
2. Log aggregation is also not working in YARN , don't know why , as I
don't have logs
3. Other issue is when i do first time flex up then it launches 2 node
managers instead of one.
Please let me know if u know about those issues.
Thanks,
Mayank
On Mon, Feb 22, 2016 at 4:21 PM, Santosh Marella <sm...@maprtech.com>
wrote:
> Hey guys.. the flow of things is as below:
>
> 1. Myriad registers itself as a Mesos framework. (Frameworks view on Mesos
> UI should show this)
> 2. Myriad by default attempts to launch 1 medium profile NM as soon as RM
> process is up.
> This is queued up until Myriad receives an offer from Mesos that is big
> enough to launch a medium profle NM.
> (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad
> attempted to launch a NM).
>
> Mayank - in your cluster, does Mesos UI show Myriad as a registered
> framework? Does it show if Myriad launched NM tasks? Perhaps a good
> starting point is to check what are the sizes of the offers Myriad is
> receiving (Mesos' master log should show this). You can also enable DEBUG
> logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's
> log4j.properties.
>
>
>
> Santosh
>
> On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier <
> b.hagemeier@fz-juelich.de
> > wrote:
>
> > Hi Mayank,
> >
> > it seems you and I are having similar issues. Let's see whether they can
> > be traced to the same causes.
> >
> >
> > Best regards,
> > Björn
> >
> > Am 22.02.2016 um 19:33 schrieb Mayank Bansal:
> > > Hi,
> > >
> > > I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x
> > >
> > >
> > > I have followed all the instructions in the wiki
> > >
> > > So the resource manager is up and i am able to launch the web UI of
> > myriad
> > > as well however whenever I try to use flex up it launches some tasks
> and
> > > put them in to pending queue and after that nothing launches.
> > >
> > > I am attaching below the resourcemanager.out and log snippets.
> > >
> > > Please let me know if I am missing something.
> > >
> > > Thanks,
> > >
> > >
> > > *I see some exceptions in resourcemanager.out files those are as
> follows*
> > >
> > >
> > > Feb 22, 2016 6:21:42 PM
> > >
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> > > resolve
> > >
> > > SEVERE: null
> > >
> > > java.lang.IllegalAccessException: Class
> > >
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> > > can not access a member of class javax.ws.rs.core.Response with
> modifiers
> > > "protected"
> > >
> > > at
> sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
> > >
> > > at java.lang.Class.newInstance(Class.java:436)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
> > >
> > > at
> > > com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89)
> > >
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >
> > > at
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > >
> > > at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > >
> > > at
> > >
> >
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> > >
> > > at
> > >
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> > >
> > > at
> > >
> >
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> > >
> > > at
> > >
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> > >
> > > at
> > >
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> > >
> > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> > >
> > > at
> > >
> >
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
> > >
> > > at
> > >
> >
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
> > >
> > > at
> > >
> >
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> > >
> > > at
> > >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
> > >
> > > at
> > >
> >
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> > >
> > > at
> > > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> > >
> > > at
> > >
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> > >
> > > at
> > >
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> > >
> > > at
> > >
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> > >
> > > at
> > >
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> > >
> > > at org.mortbay.jetty.Server.handle(Server.java:326)
> > >
> > > at
> > > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> > >
> > > at
> > >
> >
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> > >
> > > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> > >
> > > at
> > org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> > >
> > > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> > >
> > > at
> > >
> >
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> > >
> > > at
> > >
> >
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> > >
> > >
> > > Feb 22, 2016 6:21:42 PM
> > > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> > > attachTypes
> > >
> > > INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
> > >
> > > Feb 22, 2016 6:21:42 PM
> > > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> > > attachTypes
> > >
> > > INFO: Couldn't find JAX-B element for class
> > > org.apache.myriad.configuration.MyriadConfiguration
> > >
> > > Feb 22, 2016 6:21:42 PM
> > > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> > > attachTypes
> > >
> > > INFO: Couldn't find JAX-B element for class
> > > org.apache.myriad.api.model.GetSchedulerStateResponse
> > >
> > > Feb 22, 2016 6:21:42 PM
> > > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> > > attachTypes
> > >
> > > INFO: Couldn't find JAX-B element for class
> > > org.apache.myriad.api.model.FlexUpServiceRequest
> > >
> > > Feb 22, 2016 6:21:42 PM
> > >
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> > > resolve
> > >
> > > SEVERE: null
> > >
> > > E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on fd=231:
> > > Transport endpoint is not connected [107]
> > >
> > > E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on fd=231:
> > > Transport endpoint is not connected [107]
> > >
> > > E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on fd=231:
> > > Transport endpoint is not connected [107]
> > >
> > > *This is the resource manager logs*
> > >
> > > 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server
> > > listener on 8033: starting
> > >
> > > 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource:
> > > Received flexup request. Profile: medium, Instances: 1, Constraints:
> null
> > >
> > > 2016-02-22 17:39:31,266 INFO
> > org.apache.myriad.scheduler.MyriadOperations:
> > > Adding 1 NM instances to cluster
> > >
> > > 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState:
> > Marked
> > > taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size of
> > > pending queue for nm is: 1
> > >
> > > 2016-02-22 17:47:25,450 INFO
> > >
> >
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
> > > Release request cache is cleaned up
> > >
> > > 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource:
> > > Received flexup request. Profile: medium, Instances: 1, Constraints:
> null
> > >
> > > 2016-02-22 18:22:22,094 INFO
> > org.apache.myriad.scheduler.MyriadOperations:
> > > Adding 1 NM instances to cluster
> > >
> > > 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState:
> > Marked
> > > taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size of
> > > pending queue for nm is: 2
> > >
> > >
> > >
> > > Thanks,
> > >
> >
> >
> > --
> > Dipl.-Inform. Björn Hagemeier
> > Federated Systems and Data
> > Juelich Supercomputing Centre
> > Institute for Advanced Simulation
> >
> > Phone: +49 2461 61 1584
> > Fax : +49 2461 61 6656
> > Email: b.hagemeier@fz-juelich.de
> > Skype: bhagemeier
> > WWW : http://www.fz-juelich.de/jsc
> >
> > JSC is the coordinator of the
> > John von Neumann Institute for Computing
> > and member of the
> > Gauss Centre for Supercomputing
> >
> >
> >
> -------------------------------------------------------------------------------------
> >
> >
> -------------------------------------------------------------------------------------
> > Forschungszentrum Juelich GmbH
> > 52425 Juelich
> > Sitz der Gesellschaft: Juelich
> > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> > Prof. Dr. Sebastian M. Schmidt
> >
> >
> -------------------------------------------------------------------------------------
> >
> >
> -------------------------------------------------------------------------------------
> >
> >
>
--
- Mayank
Re: Not able to launch Node Managers via Myriad
Posted by Aashreya Shankar <as...@maprtech.com>.
Please check the white listed users in the following properties file :
HADOOP_HOME/etc/hadoop/container-executor.cfg
Need to set either of these 2 properties.
min.user.id=500 (make it < 99)
allowed.system.users=mapr (or add user here)
Need to give permissions for the user to run the job.
Thank you
Aashreya
On Fri, Feb 26, 2016 at 12:22 AM, Björn Hagemeier <b.hagemeier@fz-juelich.de
> wrote:
> Dear Santosh,
>
> thanks a lot for having me check the myriad-config-default.yml. I did
> not have my own configuration and due to a lack of warnings in any of
> the logs, I had assumed everything to be ok to some extent.
>
> So, now I have the myriad-config-default.yml with much of the needed
> configuration. On the way I ran into an issue also mentioned in [1], so
> I have set JAVA_HOME and YARN_NODEMANAGER_OPTS now as well as
> HADOOP_CONF_DIR and HADOOP_LIBEXEC_DIR, which are used by the NM startup
> scripts.
>
> I can now see NMs being started and torn down as a result of flexing
> up/down from the Myriad UI. However, when sending Yarn jobs, e.g.
> terasort, I am having issues with the user mapping.
>
> I get the following message:
>
> ==================================================
> Diagnostics: Application application_1456472909573_0002 initialization
> failed (exitCode=255) with output: main : command provided 0
> main : run as user is nobody
> main : requested yarn user is bjoernh
> Requested user nobody is not whitelisted and has id 99,which is below
> the minimum allowed 1000
> ==================================================
>
> When I still used my Yarn installation w/o Myriad, this worked fine.
> Could this be related to the framework user? I set it to "yarn" as this
> is that the package installation expects and has been working to the
> best of my knowledge.
>
>
> Cheers,
> Björn
>
>
> [1]
>
> https://mail-archives.apache.org/mod_mbox/myriad-dev/201512.mbox/%3CCA+kW_myt4wpr-b54kJYO0hxKWSrTHez2x41FfTCuomvmQrQwOg@mail.gmail.com%3E
>
> Am 23.02.2016 um 20:00 schrieb Santosh Marella:
> >> What makes me wonder is that I do have a running Myriad, i.e. I can
> >> access the Myriad UI on port 8192, but it does not show up in Mesos.
> >
> > It's possible you have a wrong Myriad configuration. Can you please check
> > the mesosMaster address in your myriad-config-default.yml file?
> >
> > e.g.:
> >
> https://github.com/apache/incubator-myriad/blob/master/myriad-scheduler/src/main/resources/myriad-config-default.yml#L19
> >
> > Santosh
> >
> > On Tue, Feb 23, 2016 at 3:55 AM, Björn Hagemeier <
> b.hagemeier@fz-juelich.de>
> > wrote:
> >
> >> Hi Santosh,
> >>
> >> Am 23.02.2016 um 01:21 schrieb Santosh Marella:
> >>> Hey guys.. the flow of things is as below:
> >>>
> >>> 1. Myriad registers itself as a Mesos framework. (Frameworks view on
> >> Mesos
> >>> UI should show this)
> >> I think this is where it is already failing for me. I do not see the
> >> Myriad framework in the UI.
> >>> 2. Myriad by default attempts to launch 1 medium profile NM as soon as
> RM
> >>> process is up.
> >>> This is queued up until Myriad receives an offer from Mesos that is
> >> big
> >>> enough to launch a medium profle NM.
> >> Yes, I do see the medium profile NM in the Myriad UI.
> >>> (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad
> >>> attempted to launch a NM).
> >> Nope, unfortunately it does not.
> >>
> >> What makes me wonder is that I do have a running Myriad, i.e. I can
> >> access the Myriad UI on port 8192, but it does not show up in Mesos.
> >>
> >> I guess my other problem, not seeing any offers, is a follow-on of this.
> >>
> >>>
> >>> Mayank - in your cluster, does Mesos UI show Myriad as a registered
> >>> framework? Does it show if Myriad launched NM tasks? Perhaps a good
> >>> starting point is to check what are the sizes of the offers Myriad is
> >>> receiving (Mesos' master log should show this). You can also enable
> DEBUG
> >>> logging in RM for myriad by setting "org.apache.myriad" to DEBUG in
> RM's
> >>> log4j.properties.
> >> I will try and increase the log level to DEBUG.
> >>
> >>
> >> Cheers,
> >> Björn
> >>>
> >>>
> >>>
> >>> Santosh
> >>>
> >>> On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier <
> >> b.hagemeier@fz-juelich.de
> >>>> wrote:
> >>>
> >>>> Hi Mayank,
> >>>>
> >>>> it seems you and I are having similar issues. Let's see whether they
> can
> >>>> be traced to the same causes.
> >>>>
> >>>>
> >>>> Best regards,
> >>>> Björn
> >>>>
> >>>> Am 22.02.2016 um 19:33 schrieb Mayank Bansal:
> >>>>> Hi,
> >>>>>
> >>>>> I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x
> >>>>>
> >>>>>
> >>>>> I have followed all the instructions in the wiki
> >>>>>
> >>>>> So the resource manager is up and i am able to launch the web UI of
> >>>> myriad
> >>>>> as well however whenever I try to use flex up it launches some tasks
> >> and
> >>>>> put them in to pending queue and after that nothing launches.
> >>>>>
> >>>>> I am attaching below the resourcemanager.out and log snippets.
> >>>>>
> >>>>> Please let me know if I am missing something.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>>
> >>>>> *I see some exceptions in resourcemanager.out files those are as
> >> follows*
> >>>>>
> >>>>>
> >>>>> Feb 22, 2016 6:21:42 PM
> >>>>>
> >>
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> >>>>> resolve
> >>>>>
> >>>>> SEVERE: null
> >>>>>
> >>>>> java.lang.IllegalAccessException: Class
> >>>>>
> >>
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> >>>>> can not access a member of class javax.ws.rs.core.Response with
> >> modifiers
> >>>>> "protected"
> >>>>>
> >>>>> at
> >> sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
> >>>>>
> >>>>> at java.lang.Class.newInstance(Class.java:436)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
> >>>>>
> >>>>> at
> >>>>> com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89)
> >>>>>
> >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>>
> >>>>> at java.lang.reflect.Method.invoke(Method.java:498)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> >>>>>
> >>>>> at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> >>>>>
> >>>>> at
> >>>>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> >>>>>
> >>>>> at
> >>>>>
> >> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> >>>>>
> >>>>> at
> >>>>>
> >> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> >>>>>
> >>>>> at
> >>>>>
> >> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> >>>>>
> >>>>> at org.mortbay.jetty.Server.handle(Server.java:326)
> >>>>>
> >>>>> at
> >>>>>
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> >>>>>
> >>>>> at
> org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> >>>>>
> >>>>> at
> >>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> >>>>>
> >>>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> >>>>>
> >>>>> at
> >>>>>
> >>>>
> >>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> >>>>>
> >>>>>
> >>>>> Feb 22, 2016 6:21:42 PM
> >>>>>
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> >>>>> attachTypes
> >>>>>
> >>>>> INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
> >>>>>
> >>>>> Feb 22, 2016 6:21:42 PM
> >>>>>
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> >>>>> attachTypes
> >>>>>
> >>>>> INFO: Couldn't find JAX-B element for class
> >>>>> org.apache.myriad.configuration.MyriadConfiguration
> >>>>>
> >>>>> Feb 22, 2016 6:21:42 PM
> >>>>>
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> >>>>> attachTypes
> >>>>>
> >>>>> INFO: Couldn't find JAX-B element for class
> >>>>> org.apache.myriad.api.model.GetSchedulerStateResponse
> >>>>>
> >>>>> Feb 22, 2016 6:21:42 PM
> >>>>>
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> >>>>> attachTypes
> >>>>>
> >>>>> INFO: Couldn't find JAX-B element for class
> >>>>> org.apache.myriad.api.model.FlexUpServiceRequest
> >>>>>
> >>>>> Feb 22, 2016 6:21:42 PM
> >>>>>
> >>
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> >>>>> resolve
> >>>>>
> >>>>> SEVERE: null
> >>>>>
> >>>>> E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on
> fd=231:
> >>>>> Transport endpoint is not connected [107]
> >>>>>
> >>>>> E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on
> fd=231:
> >>>>> Transport endpoint is not connected [107]
> >>>>>
> >>>>> E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on
> fd=231:
> >>>>> Transport endpoint is not connected [107]
> >>>>>
> >>>>> *This is the resource manager logs*
> >>>>>
> >>>>> 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server
> >>>>> listener on 8033: starting
> >>>>>
> >>>>> 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource:
> >>>>> Received flexup request. Profile: medium, Instances: 1, Constraints:
> >> null
> >>>>>
> >>>>> 2016-02-22 17:39:31,266 INFO
> >>>> org.apache.myriad.scheduler.MyriadOperations:
> >>>>> Adding 1 NM instances to cluster
> >>>>>
> >>>>> 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState:
> >>>> Marked
> >>>>> taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size
> of
> >>>>> pending queue for nm is: 1
> >>>>>
> >>>>> 2016-02-22 17:47:25,450 INFO
> >>>>>
> >>>>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
> >>>>> Release request cache is cleaned up
> >>>>>
> >>>>> 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource:
> >>>>> Received flexup request. Profile: medium, Instances: 1, Constraints:
> >> null
> >>>>>
> >>>>> 2016-02-22 18:22:22,094 INFO
> >>>> org.apache.myriad.scheduler.MyriadOperations:
> >>>>> Adding 1 NM instances to cluster
> >>>>>
> >>>>> 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState:
> >>>> Marked
> >>>>> taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size
> of
> >>>>> pending queue for nm is: 2
> >>>>>
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Dipl.-Inform. Björn Hagemeier
> >>>> Federated Systems and Data
> >>>> Juelich Supercomputing Centre
> >>>> Institute for Advanced Simulation
> >>>>
> >>>> Phone: +49 2461 61 1584
> >>>> Fax : +49 2461 61 6656
> >>>> Email: b.hagemeier@fz-juelich.de
> >>>> Skype: bhagemeier
> >>>> WWW : http://www.fz-juelich.de/jsc
> >>>>
> >>>> JSC is the coordinator of the
> >>>> John von Neumann Institute for Computing
> >>>> and member of the
> >>>> Gauss Centre for Supercomputing
> >>>>
> >>>>
> >>>>
> >>
> -------------------------------------------------------------------------------------
> >>>>
> >>>>
> >>
> -------------------------------------------------------------------------------------
> >>>> Forschungszentrum Juelich GmbH
> >>>> 52425 Juelich
> >>>> Sitz der Gesellschaft: Juelich
> >>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> >>>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> >>>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> >>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> >>>> Prof. Dr. Sebastian M. Schmidt
> >>>>
> >>>>
> >>
> -------------------------------------------------------------------------------------
> >>>>
> >>>>
> >>
> -------------------------------------------------------------------------------------
> >>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> Dipl.-Inform. Björn Hagemeier
> >> Federated Systems and Data
> >> Juelich Supercomputing Centre
> >> Institute for Advanced Simulation
> >>
> >> Phone: +49 2461 61 1584
> >> Fax : +49 2461 61 6656
> >> Email: b.hagemeier@fz-juelich.de
> >> Skype: bhagemeier
> >> WWW : http://www.fz-juelich.de/jsc
> >>
> >> JSC is the coordinator of the
> >> John von Neumann Institute for Computing
> >> and member of the
> >> Gauss Centre for Supercomputing
> >>
> >>
> >>
> -------------------------------------------------------------------------------------
> >>
> >>
> -------------------------------------------------------------------------------------
> >> Forschungszentrum Juelich GmbH
> >> 52425 Juelich
> >> Sitz der Gesellschaft: Juelich
> >> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> >> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> >> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> >> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> >> Prof. Dr. Sebastian M. Schmidt
> >>
> >>
> -------------------------------------------------------------------------------------
> >>
> >>
> -------------------------------------------------------------------------------------
> >>
> >>
> >
>
>
> --
> Dipl.-Inform. Björn Hagemeier
> Federated Systems and Data
> Juelich Supercomputing Centre
> Institute for Advanced Simulation
>
> Phone: +49 2461 61 1584
> Fax : +49 2461 61 6656
> Email: b.hagemeier@fz-juelich.de
> Skype: bhagemeier
> WWW : http://www.fz-juelich.de/jsc
>
> JSC is the coordinator of the
> John von Neumann Institute for Computing
> and member of the
> Gauss Centre for Supercomputing
>
>
> -------------------------------------------------------------------------------------
>
> -------------------------------------------------------------------------------------
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
>
> -------------------------------------------------------------------------------------
>
> -------------------------------------------------------------------------------------
>
>
>
Re: Not able to launch Node Managers via Myriad
Posted by Björn Hagemeier <b....@fz-juelich.de>.
Dear Santosh,
thanks a lot for having me check the myriad-config-default.yml. I did
not have my own configuration and due to a lack of warnings in any of
the logs, I had assumed everything to be ok to some extent.
So, now I have the myriad-config-default.yml with much of the needed
configuration. On the way I ran into an issue also mentioned in [1], so
I have set JAVA_HOME and YARN_NODEMANAGER_OPTS now as well as
HADOOP_CONF_DIR and HADOOP_LIBEXEC_DIR, which are used by the NM startup
scripts.
I can now see NMs being started and torn down as a result of flexing
up/down from the Myriad UI. However, when sending Yarn jobs, e.g.
terasort, I am having issues with the user mapping.
I get the following message:
==================================================
Diagnostics: Application application_1456472909573_0002 initialization
failed (exitCode=255) with output: main : command provided 0
main : run as user is nobody
main : requested yarn user is bjoernh
Requested user nobody is not whitelisted and has id 99,which is below
the minimum allowed 1000
==================================================
When I still used my Yarn installation w/o Myriad, this worked fine.
Could this be related to the framework user? I set it to "yarn" as this
is that the package installation expects and has been working to the
best of my knowledge.
Cheers,
Björn
[1]
https://mail-archives.apache.org/mod_mbox/myriad-dev/201512.mbox/%3CCA+kW_myt4wpr-b54kJYO0hxKWSrTHez2x41FfTCuomvmQrQwOg@mail.gmail.com%3E
Am 23.02.2016 um 20:00 schrieb Santosh Marella:
>> What makes me wonder is that I do have a running Myriad, i.e. I can
>> access the Myriad UI on port 8192, but it does not show up in Mesos.
>
> It's possible you have a wrong Myriad configuration. Can you please check
> the mesosMaster address in your myriad-config-default.yml file?
>
> e.g.:
> https://github.com/apache/incubator-myriad/blob/master/myriad-scheduler/src/main/resources/myriad-config-default.yml#L19
>
> Santosh
>
> On Tue, Feb 23, 2016 at 3:55 AM, Björn Hagemeier <b....@fz-juelich.de>
> wrote:
>
>> Hi Santosh,
>>
>> Am 23.02.2016 um 01:21 schrieb Santosh Marella:
>>> Hey guys.. the flow of things is as below:
>>>
>>> 1. Myriad registers itself as a Mesos framework. (Frameworks view on
>> Mesos
>>> UI should show this)
>> I think this is where it is already failing for me. I do not see the
>> Myriad framework in the UI.
>>> 2. Myriad by default attempts to launch 1 medium profile NM as soon as RM
>>> process is up.
>>> This is queued up until Myriad receives an offer from Mesos that is
>> big
>>> enough to launch a medium profle NM.
>> Yes, I do see the medium profile NM in the Myriad UI.
>>> (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad
>>> attempted to launch a NM).
>> Nope, unfortunately it does not.
>>
>> What makes me wonder is that I do have a running Myriad, i.e. I can
>> access the Myriad UI on port 8192, but it does not show up in Mesos.
>>
>> I guess my other problem, not seeing any offers, is a follow-on of this.
>>
>>>
>>> Mayank - in your cluster, does Mesos UI show Myriad as a registered
>>> framework? Does it show if Myriad launched NM tasks? Perhaps a good
>>> starting point is to check what are the sizes of the offers Myriad is
>>> receiving (Mesos' master log should show this). You can also enable DEBUG
>>> logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's
>>> log4j.properties.
>> I will try and increase the log level to DEBUG.
>>
>>
>> Cheers,
>> Björn
>>>
>>>
>>>
>>> Santosh
>>>
>>> On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier <
>> b.hagemeier@fz-juelich.de
>>>> wrote:
>>>
>>>> Hi Mayank,
>>>>
>>>> it seems you and I are having similar issues. Let's see whether they can
>>>> be traced to the same causes.
>>>>
>>>>
>>>> Best regards,
>>>> Björn
>>>>
>>>> Am 22.02.2016 um 19:33 schrieb Mayank Bansal:
>>>>> Hi,
>>>>>
>>>>> I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x
>>>>>
>>>>>
>>>>> I have followed all the instructions in the wiki
>>>>>
>>>>> So the resource manager is up and i am able to launch the web UI of
>>>> myriad
>>>>> as well however whenever I try to use flex up it launches some tasks
>> and
>>>>> put them in to pending queue and after that nothing launches.
>>>>>
>>>>> I am attaching below the resourcemanager.out and log snippets.
>>>>>
>>>>> Please let me know if I am missing something.
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>> *I see some exceptions in resourcemanager.out files those are as
>> follows*
>>>>>
>>>>>
>>>>> Feb 22, 2016 6:21:42 PM
>>>>>
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>>> resolve
>>>>>
>>>>> SEVERE: null
>>>>>
>>>>> java.lang.IllegalAccessException: Class
>>>>>
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>>> can not access a member of class javax.ws.rs.core.Response with
>> modifiers
>>>>> "protected"
>>>>>
>>>>> at
>> sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
>>>>>
>>>>> at java.lang.Class.newInstance(Class.java:436)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
>>>>>
>>>>> at
>>>>> com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89)
>>>>>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>
>>>>> at
>>>>>
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>
>>>>> at
>>>>>
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>>>>>
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>>>>>
>>>>> at
>>>>>
>>>>
>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>>>>>
>>>>> at
>>>>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>>>>>
>>>>> at
>>>>>
>>>>
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>>
>>>>> at
>>>>>
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>>
>>>>> at
>>>>>
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>>
>>>>> at
>>>>>
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>>
>>>>> at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>>
>>>>> at
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>>
>>>>> at
>>>>>
>>>>
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>>
>>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>>
>>>>> at
>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>>
>>>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>>
>>>>> at
>>>>>
>>>>
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>>
>>>>> at
>>>>>
>>>>
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>>
>>>>>
>>>>> Feb 22, 2016 6:21:42 PM
>>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>>> attachTypes
>>>>>
>>>>> INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
>>>>>
>>>>> Feb 22, 2016 6:21:42 PM
>>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>>> attachTypes
>>>>>
>>>>> INFO: Couldn't find JAX-B element for class
>>>>> org.apache.myriad.configuration.MyriadConfiguration
>>>>>
>>>>> Feb 22, 2016 6:21:42 PM
>>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>>> attachTypes
>>>>>
>>>>> INFO: Couldn't find JAX-B element for class
>>>>> org.apache.myriad.api.model.GetSchedulerStateResponse
>>>>>
>>>>> Feb 22, 2016 6:21:42 PM
>>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>>> attachTypes
>>>>>
>>>>> INFO: Couldn't find JAX-B element for class
>>>>> org.apache.myriad.api.model.FlexUpServiceRequest
>>>>>
>>>>> Feb 22, 2016 6:21:42 PM
>>>>>
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>>> resolve
>>>>>
>>>>> SEVERE: null
>>>>>
>>>>> E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on fd=231:
>>>>> Transport endpoint is not connected [107]
>>>>>
>>>>> E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on fd=231:
>>>>> Transport endpoint is not connected [107]
>>>>>
>>>>> E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on fd=231:
>>>>> Transport endpoint is not connected [107]
>>>>>
>>>>> *This is the resource manager logs*
>>>>>
>>>>> 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> listener on 8033: starting
>>>>>
>>>>> 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource:
>>>>> Received flexup request. Profile: medium, Instances: 1, Constraints:
>> null
>>>>>
>>>>> 2016-02-22 17:39:31,266 INFO
>>>> org.apache.myriad.scheduler.MyriadOperations:
>>>>> Adding 1 NM instances to cluster
>>>>>
>>>>> 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState:
>>>> Marked
>>>>> taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size of
>>>>> pending queue for nm is: 1
>>>>>
>>>>> 2016-02-22 17:47:25,450 INFO
>>>>>
>>>>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
>>>>> Release request cache is cleaned up
>>>>>
>>>>> 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource:
>>>>> Received flexup request. Profile: medium, Instances: 1, Constraints:
>> null
>>>>>
>>>>> 2016-02-22 18:22:22,094 INFO
>>>> org.apache.myriad.scheduler.MyriadOperations:
>>>>> Adding 1 NM instances to cluster
>>>>>
>>>>> 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState:
>>>> Marked
>>>>> taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size of
>>>>> pending queue for nm is: 2
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>
>>>>
>>>> --
>>>> Dipl.-Inform. Björn Hagemeier
>>>> Federated Systems and Data
>>>> Juelich Supercomputing Centre
>>>> Institute for Advanced Simulation
>>>>
>>>> Phone: +49 2461 61 1584
>>>> Fax : +49 2461 61 6656
>>>> Email: b.hagemeier@fz-juelich.de
>>>> Skype: bhagemeier
>>>> WWW : http://www.fz-juelich.de/jsc
>>>>
>>>> JSC is the coordinator of the
>>>> John von Neumann Institute for Computing
>>>> and member of the
>>>> Gauss Centre for Supercomputing
>>>>
>>>>
>>>>
>> -------------------------------------------------------------------------------------
>>>>
>>>>
>> -------------------------------------------------------------------------------------
>>>> Forschungszentrum Juelich GmbH
>>>> 52425 Juelich
>>>> Sitz der Gesellschaft: Juelich
>>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>>>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>>>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>>>> Prof. Dr. Sebastian M. Schmidt
>>>>
>>>>
>> -------------------------------------------------------------------------------------
>>>>
>>>>
>> -------------------------------------------------------------------------------------
>>>>
>>>>
>>>
>>
>>
>> --
>> Dipl.-Inform. Björn Hagemeier
>> Federated Systems and Data
>> Juelich Supercomputing Centre
>> Institute for Advanced Simulation
>>
>> Phone: +49 2461 61 1584
>> Fax : +49 2461 61 6656
>> Email: b.hagemeier@fz-juelich.de
>> Skype: bhagemeier
>> WWW : http://www.fz-juelich.de/jsc
>>
>> JSC is the coordinator of the
>> John von Neumann Institute for Computing
>> and member of the
>> Gauss Centre for Supercomputing
>>
>>
>> -------------------------------------------------------------------------------------
>>
>> -------------------------------------------------------------------------------------
>> Forschungszentrum Juelich GmbH
>> 52425 Juelich
>> Sitz der Gesellschaft: Juelich
>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>> Prof. Dr. Sebastian M. Schmidt
>>
>> -------------------------------------------------------------------------------------
>>
>> -------------------------------------------------------------------------------------
>>
>>
>
--
Dipl.-Inform. Björn Hagemeier
Federated Systems and Data
Juelich Supercomputing Centre
Institute for Advanced Simulation
Phone: +49 2461 61 1584
Fax : +49 2461 61 6656
Email: b.hagemeier@fz-juelich.de
Skype: bhagemeier
WWW : http://www.fz-juelich.de/jsc
JSC is the coordinator of the
John von Neumann Institute for Computing
and member of the
Gauss Centre for Supercomputing
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Re: Not able to launch Node Managers via Myriad
Posted by Santosh Marella <sm...@maprtech.com>.
>What makes me wonder is that I do have a running Myriad, i.e. I can
>access the Myriad UI on port 8192, but it does not show up in Mesos.
It's possible you have a wrong Myriad configuration. Can you please check
the mesosMaster address in your myriad-config-default.yml file?
e.g.:
https://github.com/apache/incubator-myriad/blob/master/myriad-scheduler/src/main/resources/myriad-config-default.yml#L19
Santosh
On Tue, Feb 23, 2016 at 3:55 AM, Björn Hagemeier <b....@fz-juelich.de>
wrote:
> Hi Santosh,
>
> Am 23.02.2016 um 01:21 schrieb Santosh Marella:
> > Hey guys.. the flow of things is as below:
> >
> > 1. Myriad registers itself as a Mesos framework. (Frameworks view on
> Mesos
> > UI should show this)
> I think this is where it is already failing for me. I do not see the
> Myriad framework in the UI.
> > 2. Myriad by default attempts to launch 1 medium profile NM as soon as RM
> > process is up.
> > This is queued up until Myriad receives an offer from Mesos that is
> big
> > enough to launch a medium profle NM.
> Yes, I do see the medium profile NM in the Myriad UI.
> > (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad
> > attempted to launch a NM).
> Nope, unfortunately it does not.
>
> What makes me wonder is that I do have a running Myriad, i.e. I can
> access the Myriad UI on port 8192, but it does not show up in Mesos.
>
> I guess my other problem, not seeing any offers, is a follow-on of this.
>
> >
> > Mayank - in your cluster, does Mesos UI show Myriad as a registered
> > framework? Does it show if Myriad launched NM tasks? Perhaps a good
> > starting point is to check what are the sizes of the offers Myriad is
> > receiving (Mesos' master log should show this). You can also enable DEBUG
> > logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's
> > log4j.properties.
> I will try and increase the log level to DEBUG.
>
>
> Cheers,
> Björn
> >
> >
> >
> > Santosh
> >
> > On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier <
> b.hagemeier@fz-juelich.de
> >> wrote:
> >
> >> Hi Mayank,
> >>
> >> it seems you and I are having similar issues. Let's see whether they can
> >> be traced to the same causes.
> >>
> >>
> >> Best regards,
> >> Björn
> >>
> >> Am 22.02.2016 um 19:33 schrieb Mayank Bansal:
> >>> Hi,
> >>>
> >>> I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x
> >>>
> >>>
> >>> I have followed all the instructions in the wiki
> >>>
> >>> So the resource manager is up and i am able to launch the web UI of
> >> myriad
> >>> as well however whenever I try to use flex up it launches some tasks
> and
> >>> put them in to pending queue and after that nothing launches.
> >>>
> >>> I am attaching below the resourcemanager.out and log snippets.
> >>>
> >>> Please let me know if I am missing something.
> >>>
> >>> Thanks,
> >>>
> >>>
> >>> *I see some exceptions in resourcemanager.out files those are as
> follows*
> >>>
> >>>
> >>> Feb 22, 2016 6:21:42 PM
> >>>
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> >>> resolve
> >>>
> >>> SEVERE: null
> >>>
> >>> java.lang.IllegalAccessException: Class
> >>>
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> >>> can not access a member of class javax.ws.rs.core.Response with
> modifiers
> >>> "protected"
> >>>
> >>> at
> sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
> >>>
> >>> at java.lang.Class.newInstance(Class.java:436)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
> >>>
> >>> at
> >>> com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89)
> >>>
> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>
> >>> at
> >>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>>
> >>> at
> >>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>
> >>> at java.lang.reflect.Method.invoke(Method.java:498)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> >>>
> >>> at
> >>>
> >>
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> >>>
> >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> >>>
> >>> at
> >>>
> >>
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
> >>>
> >>> at
> >>>
> >>
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
> >>>
> >>> at
> >>>
> >>
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> >>>
> >>> at
> >>>
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
> >>>
> >>> at
> >>>
> >>
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> >>>
> >>> at
> >>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> >>>
> >>> at
> >>>
> >>
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> >>>
> >>> at
> >>>
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> >>>
> >>> at
> >>>
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> >>>
> >>> at
> >>>
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> >>>
> >>> at org.mortbay.jetty.Server.handle(Server.java:326)
> >>>
> >>> at
> >>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> >>>
> >>> at
> >>>
> >>
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> >>>
> >>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> >>>
> >>> at
> >> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> >>>
> >>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> >>>
> >>> at
> >>>
> >>
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> >>>
> >>> at
> >>>
> >>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> >>>
> >>>
> >>> Feb 22, 2016 6:21:42 PM
> >>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> >>> attachTypes
> >>>
> >>> INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
> >>>
> >>> Feb 22, 2016 6:21:42 PM
> >>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> >>> attachTypes
> >>>
> >>> INFO: Couldn't find JAX-B element for class
> >>> org.apache.myriad.configuration.MyriadConfiguration
> >>>
> >>> Feb 22, 2016 6:21:42 PM
> >>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> >>> attachTypes
> >>>
> >>> INFO: Couldn't find JAX-B element for class
> >>> org.apache.myriad.api.model.GetSchedulerStateResponse
> >>>
> >>> Feb 22, 2016 6:21:42 PM
> >>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> >>> attachTypes
> >>>
> >>> INFO: Couldn't find JAX-B element for class
> >>> org.apache.myriad.api.model.FlexUpServiceRequest
> >>>
> >>> Feb 22, 2016 6:21:42 PM
> >>>
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> >>> resolve
> >>>
> >>> SEVERE: null
> >>>
> >>> E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on fd=231:
> >>> Transport endpoint is not connected [107]
> >>>
> >>> E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on fd=231:
> >>> Transport endpoint is not connected [107]
> >>>
> >>> E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on fd=231:
> >>> Transport endpoint is not connected [107]
> >>>
> >>> *This is the resource manager logs*
> >>>
> >>> 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server
> >>> listener on 8033: starting
> >>>
> >>> 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource:
> >>> Received flexup request. Profile: medium, Instances: 1, Constraints:
> null
> >>>
> >>> 2016-02-22 17:39:31,266 INFO
> >> org.apache.myriad.scheduler.MyriadOperations:
> >>> Adding 1 NM instances to cluster
> >>>
> >>> 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState:
> >> Marked
> >>> taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size of
> >>> pending queue for nm is: 1
> >>>
> >>> 2016-02-22 17:47:25,450 INFO
> >>>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
> >>> Release request cache is cleaned up
> >>>
> >>> 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource:
> >>> Received flexup request. Profile: medium, Instances: 1, Constraints:
> null
> >>>
> >>> 2016-02-22 18:22:22,094 INFO
> >> org.apache.myriad.scheduler.MyriadOperations:
> >>> Adding 1 NM instances to cluster
> >>>
> >>> 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState:
> >> Marked
> >>> taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size of
> >>> pending queue for nm is: 2
> >>>
> >>>
> >>>
> >>> Thanks,
> >>>
> >>
> >>
> >> --
> >> Dipl.-Inform. Björn Hagemeier
> >> Federated Systems and Data
> >> Juelich Supercomputing Centre
> >> Institute for Advanced Simulation
> >>
> >> Phone: +49 2461 61 1584
> >> Fax : +49 2461 61 6656
> >> Email: b.hagemeier@fz-juelich.de
> >> Skype: bhagemeier
> >> WWW : http://www.fz-juelich.de/jsc
> >>
> >> JSC is the coordinator of the
> >> John von Neumann Institute for Computing
> >> and member of the
> >> Gauss Centre for Supercomputing
> >>
> >>
> >>
> -------------------------------------------------------------------------------------
> >>
> >>
> -------------------------------------------------------------------------------------
> >> Forschungszentrum Juelich GmbH
> >> 52425 Juelich
> >> Sitz der Gesellschaft: Juelich
> >> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> >> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> >> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> >> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> >> Prof. Dr. Sebastian M. Schmidt
> >>
> >>
> -------------------------------------------------------------------------------------
> >>
> >>
> -------------------------------------------------------------------------------------
> >>
> >>
> >
>
>
> --
> Dipl.-Inform. Björn Hagemeier
> Federated Systems and Data
> Juelich Supercomputing Centre
> Institute for Advanced Simulation
>
> Phone: +49 2461 61 1584
> Fax : +49 2461 61 6656
> Email: b.hagemeier@fz-juelich.de
> Skype: bhagemeier
> WWW : http://www.fz-juelich.de/jsc
>
> JSC is the coordinator of the
> John von Neumann Institute for Computing
> and member of the
> Gauss Centre for Supercomputing
>
>
> -------------------------------------------------------------------------------------
>
> -------------------------------------------------------------------------------------
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
>
> -------------------------------------------------------------------------------------
>
> -------------------------------------------------------------------------------------
>
>
Re: Not able to launch Node Managers via Myriad
Posted by yuliya Feldman <yu...@yahoo.com.INVALID>.
Could you provide your yarn-site.xml and myriad-config-default.yml ?
Let's make sure you have following in your yarn-site.xml:
<property> <name>yarn.resourcemanager.scheduler.class</name> <value>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler</value> <description>One can configure other scehdulers as well from following list: com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler, com.ebay.myriad.scheduler.yarn.MyriadFifoScheduler</description> </property> <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>${nodemanager.resource.cpu-vcores}</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>${nodemanager.resource.memory-mb}</value> </property> <property> <name>yarn.nodemanager.address</name> <value>${myriad.yarn.nodemanager.address}</value> </property> <property> <name>yarn.nodemanager.webapp.address</name> <value>${myriad.yarn.nodemanager.webapp.address}</value> </property> <property> <name>yarn.nodemanager.webapp.https.address</name> <value>${myriad.yarn.nodemanager.webapp.address}</value> </property> <property> <name>yarn.nodemanager.localizer.address</name> <value>${myriad.yarn.nodemanager.localizer.address}</value> </property> <property> <name>mapreduce.shuffle.port</name> <value>${myriad.mapreduce.shuffle.port}</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle,myriad_executor</value> </property> <property> <name>yarn.nodemanager.aux-services.myriad_executor.class</name> <value>com.ebay.myriad.executor.MyriadExecutorAuxService</value> </property>
And in your yml file (should be at least linked to */etc/hadoop place)You have following line defined correctly:
mesosMaster: zk://<zk_address>/mesos
Thanks,Yuliya
From: Björn Hagemeier <b....@fz-juelich.de>
To: dev@myriad.incubator.apache.org
Sent: Tuesday, February 23, 2016 6:21 AM
Subject: Re: Not able to launch Node Managers via Myriad
Hi all,
Am 23.02.2016 um 12:55 schrieb Björn Hagemeier:
> Hi Santosh,
>
> Am 23.02.2016 um 01:21 schrieb Santosh Marella:
>> Hey guys.. the flow of things is as below:
>>
>> 1. Myriad registers itself as a Mesos framework. (Frameworks view on Mesos
>> UI should show this)
does Myriad register itself as a Mesos framework automatically, just by
starting the RM as one would do it without Mesos? Or do I need to start
the RM as a Mesos framework to make the magic happen, i.e. by "mesos
execute"?
In the latter case, I think I found out why my Myriad does not seem to
do anything. I did not start the resource manager as a Mesos framework.
I somehow reckoned, the myriad integration would take care of that by
itself. When I do this, I do see the framework, but with classpath
issues (missing myriad for some reason).
I'll keep digging in that direction.
Cheers,
Björn
> I think this is where it is already failing for me. I do not see the
> Myriad framework in the UI.
>> 2. Myriad by default attempts to launch 1 medium profile NM as soon as RM
>> process is up.
>> This is queued up until Myriad receives an offer from Mesos that is big
>> enough to launch a medium profle NM.
> Yes, I do see the medium profile NM in the Myriad UI.
>> (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad
>> attempted to launch a NM).
> Nope, unfortunately it does not.
>
> What makes me wonder is that I do have a running Myriad, i.e. I can
> access the Myriad UI on port 8192, but it does not show up in Mesos.
>
> I guess my other problem, not seeing any offers, is a follow-on of this.
>
>>
>> Mayank - in your cluster, does Mesos UI show Myriad as a registered
>> framework? Does it show if Myriad launched NM tasks? Perhaps a good
>> starting point is to check what are the sizes of the offers Myriad is
>> receiving (Mesos' master log should show this). You can also enable DEBUG
>> logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's
>> log4j.properties.
> I will try and increase the log level to DEBUG.
>
>
> Cheers,
> Björn
>>
>>
>>
>> Santosh
>>
>> On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier <b.hagemeier@fz-juelich.de
>>> wrote:
>>
>>> Hi Mayank,
>>>
>>> it seems you and I are having similar issues. Let's see whether they can
>>> be traced to the same causes.
>>>
>>>
>>> Best regards,
>>> Björn
>>>
>>> Am 22.02.2016 um 19:33 schrieb Mayank Bansal:
>>>> Hi,
>>>>
>>>> I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x
>>>>
>>>>
>>>> I have followed all the instructions in the wiki
>>>>
>>>> So the resource manager is up and i am able to launch the web UI of
>>> myriad
>>>> as well however whenever I try to use flex up it launches some tasks and
>>>> put them in to pending queue and after that nothing launches.
>>>>
>>>> I am attaching below the resourcemanager.out and log snippets.
>>>>
>>>> Please let me know if I am missing something.
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> *I see some exceptions in resourcemanager.out files those are as follows*
>>>>
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>> resolve
>>>>
>>>> SEVERE: null
>>>>
>>>> java.lang.IllegalAccessException: Class
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>> can not access a member of class javax.ws.rs.core.Response with modifiers
>>>> "protected"
>>>>
>>>> at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
>>>>
>>>> at java.lang.Class.newInstance(Class.java:436)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
>>>>
>>>> at
>>>> com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89)
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>
>>>> at
>>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>
>>>> at
>>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>>>>
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>>>>
>>>> at
>>>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>>>>
>>>> at
>>>>
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>
>>>> at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>
>>>> at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>
>>>> at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>
>>>> at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>
>>>> at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>
>>>> at
>>>>
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>
>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>
>>>> at
>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>
>>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>
>>>> at
>>>>
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>
>>>> at
>>>>
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> attachTypes
>>>>
>>>> INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> attachTypes
>>>>
>>>> INFO: Couldn't find JAX-B element for class
>>>> org.apache.myriad.configuration.MyriadConfiguration
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> attachTypes
>>>>
>>>> INFO: Couldn't find JAX-B element for class
>>>> org.apache.myriad.api.model.GetSchedulerStateResponse
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> attachTypes
>>>>
>>>> INFO: Couldn't find JAX-B element for class
>>>> org.apache.myriad.api.model.FlexUpServiceRequest
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>> resolve
>>>>
>>>> SEVERE: null
>>>>
>>>> E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on fd=231:
>>>> Transport endpoint is not connected [107]
>>>>
>>>> E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on fd=231:
>>>> Transport endpoint is not connected [107]
>>>>
>>>> E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on fd=231:
>>>> Transport endpoint is not connected [107]
>>>>
>>>> *This is the resource manager logs*
>>>>
>>>> 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> listener on 8033: starting
>>>>
>>>> 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource:
>>>> Received flexup request. Profile: medium, Instances: 1, Constraints: null
>>>>
>>>> 2016-02-22 17:39:31,266 INFO
>>> org.apache.myriad.scheduler.MyriadOperations:
>>>> Adding 1 NM instances to cluster
>>>>
>>>> 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState:
>>> Marked
>>>> taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size of
>>>> pending queue for nm is: 1
>>>>
>>>> 2016-02-22 17:47:25,450 INFO
>>>>
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
>>>> Release request cache is cleaned up
>>>>
>>>> 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource:
>>>> Received flexup request. Profile: medium, Instances: 1, Constraints: null
>>>>
>>>> 2016-02-22 18:22:22,094 INFO
>>> org.apache.myriad.scheduler.MyriadOperations:
>>>> Adding 1 NM instances to cluster
>>>>
>>>> 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState:
>>> Marked
>>>> taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size of
>>>> pending queue for nm is: 2
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>
>>>
>>> --
>>> Dipl.-Inform. Björn Hagemeier
>>> Federated Systems and Data
>>> Juelich Supercomputing Centre
>>> Institute for Advanced Simulation
>>>
>>> Phone: +49 2461 61 1584
>>> Fax : +49 2461 61 6656
>>> Email: b.hagemeier@fz-juelich.de
>>> Skype: bhagemeier
>>> WWW : http://www.fz-juelich.de/jsc
>>>
>>> JSC is the coordinator of the
>>> John von Neumann Institute for Computing
>>> and member of the
>>> Gauss Centre for Supercomputing
>>>
>>>
>>> -------------------------------------------------------------------------------------
>>>
>>> -------------------------------------------------------------------------------------
>>> Forschungszentrum Juelich GmbH
>>> 52425 Juelich
>>> Sitz der Gesellschaft: Juelich
>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>>> Prof. Dr. Sebastian M. Schmidt
>>>
>>> -------------------------------------------------------------------------------------
>>>
>>> -------------------------------------------------------------------------------------
>>>
>>>
>>
>
>
--
Dipl.-Inform. Björn Hagemeier
Federated Systems and Data
Juelich Supercomputing Centre
Institute for Advanced Simulation
Phone: +49 2461 61 1584
Fax : +49 2461 61 6656
Email: b.hagemeier@fz-juelich.de
Skype: bhagemeier
WWW : http://www.fz-juelich.de/jsc
JSC is the coordinator of the
John von Neumann Institute for Computing
and member of the
Gauss Centre for Supercomputing
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Re: Not able to launch Node Managers via Myriad
Posted by Björn Hagemeier <b....@fz-juelich.de>.
Hi all,
Am 23.02.2016 um 12:55 schrieb Björn Hagemeier:
> Hi Santosh,
>
> Am 23.02.2016 um 01:21 schrieb Santosh Marella:
>> Hey guys.. the flow of things is as below:
>>
>> 1. Myriad registers itself as a Mesos framework. (Frameworks view on Mesos
>> UI should show this)
does Myriad register itself as a Mesos framework automatically, just by
starting the RM as one would do it without Mesos? Or do I need to start
the RM as a Mesos framework to make the magic happen, i.e. by "mesos
execute"?
In the latter case, I think I found out why my Myriad does not seem to
do anything. I did not start the resource manager as a Mesos framework.
I somehow reckoned, the myriad integration would take care of that by
itself. When I do this, I do see the framework, but with classpath
issues (missing myriad for some reason).
I'll keep digging in that direction.
Cheers,
Björn
> I think this is where it is already failing for me. I do not see the
> Myriad framework in the UI.
>> 2. Myriad by default attempts to launch 1 medium profile NM as soon as RM
>> process is up.
>> This is queued up until Myriad receives an offer from Mesos that is big
>> enough to launch a medium profle NM.
> Yes, I do see the medium profile NM in the Myriad UI.
>> (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad
>> attempted to launch a NM).
> Nope, unfortunately it does not.
>
> What makes me wonder is that I do have a running Myriad, i.e. I can
> access the Myriad UI on port 8192, but it does not show up in Mesos.
>
> I guess my other problem, not seeing any offers, is a follow-on of this.
>
>>
>> Mayank - in your cluster, does Mesos UI show Myriad as a registered
>> framework? Does it show if Myriad launched NM tasks? Perhaps a good
>> starting point is to check what are the sizes of the offers Myriad is
>> receiving (Mesos' master log should show this). You can also enable DEBUG
>> logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's
>> log4j.properties.
> I will try and increase the log level to DEBUG.
>
>
> Cheers,
> Björn
>>
>>
>>
>> Santosh
>>
>> On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier <b.hagemeier@fz-juelich.de
>>> wrote:
>>
>>> Hi Mayank,
>>>
>>> it seems you and I are having similar issues. Let's see whether they can
>>> be traced to the same causes.
>>>
>>>
>>> Best regards,
>>> Björn
>>>
>>> Am 22.02.2016 um 19:33 schrieb Mayank Bansal:
>>>> Hi,
>>>>
>>>> I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x
>>>>
>>>>
>>>> I have followed all the instructions in the wiki
>>>>
>>>> So the resource manager is up and i am able to launch the web UI of
>>> myriad
>>>> as well however whenever I try to use flex up it launches some tasks and
>>>> put them in to pending queue and after that nothing launches.
>>>>
>>>> I am attaching below the resourcemanager.out and log snippets.
>>>>
>>>> Please let me know if I am missing something.
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> *I see some exceptions in resourcemanager.out files those are as follows*
>>>>
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>> resolve
>>>>
>>>> SEVERE: null
>>>>
>>>> java.lang.IllegalAccessException: Class
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>> can not access a member of class javax.ws.rs.core.Response with modifiers
>>>> "protected"
>>>>
>>>> at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
>>>>
>>>> at java.lang.Class.newInstance(Class.java:436)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
>>>>
>>>> at
>>>> com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89)
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>
>>>> at
>>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>
>>>> at
>>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>>>>
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>>>>
>>>> at
>>>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>>>>
>>>> at
>>>>
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>
>>>> at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>
>>>> at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>
>>>> at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>
>>>> at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>
>>>> at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>
>>>> at
>>>>
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>
>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>
>>>> at
>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>
>>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>
>>>> at
>>>>
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>
>>>> at
>>>>
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> attachTypes
>>>>
>>>> INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> attachTypes
>>>>
>>>> INFO: Couldn't find JAX-B element for class
>>>> org.apache.myriad.configuration.MyriadConfiguration
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> attachTypes
>>>>
>>>> INFO: Couldn't find JAX-B element for class
>>>> org.apache.myriad.api.model.GetSchedulerStateResponse
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> attachTypes
>>>>
>>>> INFO: Couldn't find JAX-B element for class
>>>> org.apache.myriad.api.model.FlexUpServiceRequest
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>> resolve
>>>>
>>>> SEVERE: null
>>>>
>>>> E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on fd=231:
>>>> Transport endpoint is not connected [107]
>>>>
>>>> E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on fd=231:
>>>> Transport endpoint is not connected [107]
>>>>
>>>> E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on fd=231:
>>>> Transport endpoint is not connected [107]
>>>>
>>>> *This is the resource manager logs*
>>>>
>>>> 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> listener on 8033: starting
>>>>
>>>> 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource:
>>>> Received flexup request. Profile: medium, Instances: 1, Constraints: null
>>>>
>>>> 2016-02-22 17:39:31,266 INFO
>>> org.apache.myriad.scheduler.MyriadOperations:
>>>> Adding 1 NM instances to cluster
>>>>
>>>> 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState:
>>> Marked
>>>> taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size of
>>>> pending queue for nm is: 1
>>>>
>>>> 2016-02-22 17:47:25,450 INFO
>>>>
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
>>>> Release request cache is cleaned up
>>>>
>>>> 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource:
>>>> Received flexup request. Profile: medium, Instances: 1, Constraints: null
>>>>
>>>> 2016-02-22 18:22:22,094 INFO
>>> org.apache.myriad.scheduler.MyriadOperations:
>>>> Adding 1 NM instances to cluster
>>>>
>>>> 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState:
>>> Marked
>>>> taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size of
>>>> pending queue for nm is: 2
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>
>>>
>>> --
>>> Dipl.-Inform. Björn Hagemeier
>>> Federated Systems and Data
>>> Juelich Supercomputing Centre
>>> Institute for Advanced Simulation
>>>
>>> Phone: +49 2461 61 1584
>>> Fax : +49 2461 61 6656
>>> Email: b.hagemeier@fz-juelich.de
>>> Skype: bhagemeier
>>> WWW : http://www.fz-juelich.de/jsc
>>>
>>> JSC is the coordinator of the
>>> John von Neumann Institute for Computing
>>> and member of the
>>> Gauss Centre for Supercomputing
>>>
>>>
>>> -------------------------------------------------------------------------------------
>>>
>>> -------------------------------------------------------------------------------------
>>> Forschungszentrum Juelich GmbH
>>> 52425 Juelich
>>> Sitz der Gesellschaft: Juelich
>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>>> Prof. Dr. Sebastian M. Schmidt
>>>
>>> -------------------------------------------------------------------------------------
>>>
>>> -------------------------------------------------------------------------------------
>>>
>>>
>>
>
>
--
Dipl.-Inform. Björn Hagemeier
Federated Systems and Data
Juelich Supercomputing Centre
Institute for Advanced Simulation
Phone: +49 2461 61 1584
Fax : +49 2461 61 6656
Email: b.hagemeier@fz-juelich.de
Skype: bhagemeier
WWW : http://www.fz-juelich.de/jsc
JSC is the coordinator of the
John von Neumann Institute for Computing
and member of the
Gauss Centre for Supercomputing
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Re: Not able to launch Node Managers via Myriad
Posted by Björn Hagemeier <b....@fz-juelich.de>.
Hi Santosh,
Am 23.02.2016 um 01:21 schrieb Santosh Marella:
> Hey guys.. the flow of things is as below:
>
> 1. Myriad registers itself as a Mesos framework. (Frameworks view on Mesos
> UI should show this)
I think this is where it is already failing for me. I do not see the
Myriad framework in the UI.
> 2. Myriad by default attempts to launch 1 medium profile NM as soon as RM
> process is up.
> This is queued up until Myriad receives an offer from Mesos that is big
> enough to launch a medium profle NM.
Yes, I do see the medium profile NM in the Myriad UI.
> (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad
> attempted to launch a NM).
Nope, unfortunately it does not.
What makes me wonder is that I do have a running Myriad, i.e. I can
access the Myriad UI on port 8192, but it does not show up in Mesos.
I guess my other problem, not seeing any offers, is a follow-on of this.
>
> Mayank - in your cluster, does Mesos UI show Myriad as a registered
> framework? Does it show if Myriad launched NM tasks? Perhaps a good
> starting point is to check what are the sizes of the offers Myriad is
> receiving (Mesos' master log should show this). You can also enable DEBUG
> logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's
> log4j.properties.
I will try and increase the log level to DEBUG.
Cheers,
Björn
>
>
>
> Santosh
>
> On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier <b.hagemeier@fz-juelich.de
>> wrote:
>
>> Hi Mayank,
>>
>> it seems you and I are having similar issues. Let's see whether they can
>> be traced to the same causes.
>>
>>
>> Best regards,
>> Björn
>>
>> Am 22.02.2016 um 19:33 schrieb Mayank Bansal:
>>> Hi,
>>>
>>> I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x
>>>
>>>
>>> I have followed all the instructions in the wiki
>>>
>>> So the resource manager is up and i am able to launch the web UI of
>> myriad
>>> as well however whenever I try to use flex up it launches some tasks and
>>> put them in to pending queue and after that nothing launches.
>>>
>>> I am attaching below the resourcemanager.out and log snippets.
>>>
>>> Please let me know if I am missing something.
>>>
>>> Thanks,
>>>
>>>
>>> *I see some exceptions in resourcemanager.out files those are as follows*
>>>
>>>
>>> Feb 22, 2016 6:21:42 PM
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>> resolve
>>>
>>> SEVERE: null
>>>
>>> java.lang.IllegalAccessException: Class
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>> can not access a member of class javax.ws.rs.core.Response with modifiers
>>> "protected"
>>>
>>> at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
>>>
>>> at java.lang.Class.newInstance(Class.java:436)
>>>
>>> at
>>>
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
>>>
>>> at
>>>
>> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
>>>
>>> at
>>>
>> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
>>>
>>> at
>>>
>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
>>>
>>> at
>>> com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
>>>
>>> at
>>>
>> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
>>>
>>> at
>>>
>> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>> at
>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>
>>> at
>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>
>>> at
>>>
>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>>>
>>> at
>>>
>> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>>>
>>> at
>>>
>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>>>
>>> at
>>>
>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>>>
>>> at
>>>
>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>>>
>>> at
>>>
>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>>>
>>> at
>>>
>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>>>
>>> at
>>>
>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>>>
>>> at
>>>
>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>>>
>>> at
>>>
>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>>>
>>> at
>>>
>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>>>
>>> at
>>>
>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>>>
>>> at
>>>
>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>>>
>>> at
>>>
>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>>>
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>
>>> at
>>>
>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>>>
>>> at
>>>
>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>>>
>>> at
>>>
>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>>>
>>> at
>>>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>>>
>>> at
>>>
>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>>>
>>> at
>>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>>>
>>> at
>>>
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>
>>> at
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>
>>> at
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>
>>> at
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>
>>> at org.mortbay.jetty.Server.handle(Server.java:326)
>>>
>>> at
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>
>>> at
>>>
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>
>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>
>>> at
>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>
>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>
>>> at
>>>
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>
>>> at
>>>
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>
>>>
>>> Feb 22, 2016 6:21:42 PM
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>> attachTypes
>>>
>>> INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
>>>
>>> Feb 22, 2016 6:21:42 PM
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>> attachTypes
>>>
>>> INFO: Couldn't find JAX-B element for class
>>> org.apache.myriad.configuration.MyriadConfiguration
>>>
>>> Feb 22, 2016 6:21:42 PM
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>> attachTypes
>>>
>>> INFO: Couldn't find JAX-B element for class
>>> org.apache.myriad.api.model.GetSchedulerStateResponse
>>>
>>> Feb 22, 2016 6:21:42 PM
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>> attachTypes
>>>
>>> INFO: Couldn't find JAX-B element for class
>>> org.apache.myriad.api.model.FlexUpServiceRequest
>>>
>>> Feb 22, 2016 6:21:42 PM
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>> resolve
>>>
>>> SEVERE: null
>>>
>>> E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on fd=231:
>>> Transport endpoint is not connected [107]
>>>
>>> E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on fd=231:
>>> Transport endpoint is not connected [107]
>>>
>>> E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on fd=231:
>>> Transport endpoint is not connected [107]
>>>
>>> *This is the resource manager logs*
>>>
>>> 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> listener on 8033: starting
>>>
>>> 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource:
>>> Received flexup request. Profile: medium, Instances: 1, Constraints: null
>>>
>>> 2016-02-22 17:39:31,266 INFO
>> org.apache.myriad.scheduler.MyriadOperations:
>>> Adding 1 NM instances to cluster
>>>
>>> 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState:
>> Marked
>>> taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size of
>>> pending queue for nm is: 1
>>>
>>> 2016-02-22 17:47:25,450 INFO
>>>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
>>> Release request cache is cleaned up
>>>
>>> 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource:
>>> Received flexup request. Profile: medium, Instances: 1, Constraints: null
>>>
>>> 2016-02-22 18:22:22,094 INFO
>> org.apache.myriad.scheduler.MyriadOperations:
>>> Adding 1 NM instances to cluster
>>>
>>> 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState:
>> Marked
>>> taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size of
>>> pending queue for nm is: 2
>>>
>>>
>>>
>>> Thanks,
>>>
>>
>>
>> --
>> Dipl.-Inform. Björn Hagemeier
>> Federated Systems and Data
>> Juelich Supercomputing Centre
>> Institute for Advanced Simulation
>>
>> Phone: +49 2461 61 1584
>> Fax : +49 2461 61 6656
>> Email: b.hagemeier@fz-juelich.de
>> Skype: bhagemeier
>> WWW : http://www.fz-juelich.de/jsc
>>
>> JSC is the coordinator of the
>> John von Neumann Institute for Computing
>> and member of the
>> Gauss Centre for Supercomputing
>>
>>
>> -------------------------------------------------------------------------------------
>>
>> -------------------------------------------------------------------------------------
>> Forschungszentrum Juelich GmbH
>> 52425 Juelich
>> Sitz der Gesellschaft: Juelich
>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>> Prof. Dr. Sebastian M. Schmidt
>>
>> -------------------------------------------------------------------------------------
>>
>> -------------------------------------------------------------------------------------
>>
>>
>
--
Dipl.-Inform. Björn Hagemeier
Federated Systems and Data
Juelich Supercomputing Centre
Institute for Advanced Simulation
Phone: +49 2461 61 1584
Fax : +49 2461 61 6656
Email: b.hagemeier@fz-juelich.de
Skype: bhagemeier
WWW : http://www.fz-juelich.de/jsc
JSC is the coordinator of the
John von Neumann Institute for Computing
and member of the
Gauss Centre for Supercomputing
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Re: Not able to launch Node Managers via Myriad
Posted by Santosh Marella <sm...@maprtech.com>.
Hey guys.. the flow of things is as below:
1. Myriad registers itself as a Mesos framework. (Frameworks view on Mesos
UI should show this)
2. Myriad by default attempts to launch 1 medium profile NM as soon as RM
process is up.
This is queued up until Myriad receives an offer from Mesos that is big
enough to launch a medium profle NM.
(Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad
attempted to launch a NM).
Mayank - in your cluster, does Mesos UI show Myriad as a registered
framework? Does it show if Myriad launched NM tasks? Perhaps a good
starting point is to check what are the sizes of the offers Myriad is
receiving (Mesos' master log should show this). You can also enable DEBUG
logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's
log4j.properties.
Santosh
On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier <b.hagemeier@fz-juelich.de
> wrote:
> Hi Mayank,
>
> it seems you and I are having similar issues. Let's see whether they can
> be traced to the same causes.
>
>
> Best regards,
> Björn
>
> Am 22.02.2016 um 19:33 schrieb Mayank Bansal:
> > Hi,
> >
> > I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x
> >
> >
> > I have followed all the instructions in the wiki
> >
> > So the resource manager is up and i am able to launch the web UI of
> myriad
> > as well however whenever I try to use flex up it launches some tasks and
> > put them in to pending queue and after that nothing launches.
> >
> > I am attaching below the resourcemanager.out and log snippets.
> >
> > Please let me know if I am missing something.
> >
> > Thanks,
> >
> >
> > *I see some exceptions in resourcemanager.out files those are as follows*
> >
> >
> > Feb 22, 2016 6:21:42 PM
> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> > resolve
> >
> > SEVERE: null
> >
> > java.lang.IllegalAccessException: Class
> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> > can not access a member of class javax.ws.rs.core.Response with modifiers
> > "protected"
> >
> > at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
> >
> > at java.lang.Class.newInstance(Class.java:436)
> >
> > at
> >
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
> >
> > at
> >
> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
> >
> > at
> >
> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
> >
> > at
> >
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
> >
> > at
> > com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
> >
> > at
> >
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
> >
> > at
> >
> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89)
> >
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> > at java.lang.reflect.Method.invoke(Method.java:498)
> >
> > at
> >
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> >
> > at
> >
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> >
> > at
> >
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> >
> > at
> >
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> >
> > at
> >
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> >
> > at
> >
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> >
> > at
> >
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> >
> > at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> >
> > at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> >
> > at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> >
> > at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> >
> > at
> >
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> >
> > at
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> >
> > at
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> >
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> >
> > at
> >
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
> >
> > at
> >
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
> >
> > at
> >
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> >
> > at
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
> >
> > at
> >
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> >
> > at
> > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> >
> > at
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> >
> > at
> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> >
> > at
> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> >
> > at
> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> >
> > at org.mortbay.jetty.Server.handle(Server.java:326)
> >
> > at
> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> >
> > at
> >
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> >
> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> >
> > at
> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> >
> > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> >
> > at
> >
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> >
> > at
> >
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> >
> >
> > Feb 22, 2016 6:21:42 PM
> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> > attachTypes
> >
> > INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
> >
> > Feb 22, 2016 6:21:42 PM
> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> > attachTypes
> >
> > INFO: Couldn't find JAX-B element for class
> > org.apache.myriad.configuration.MyriadConfiguration
> >
> > Feb 22, 2016 6:21:42 PM
> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> > attachTypes
> >
> > INFO: Couldn't find JAX-B element for class
> > org.apache.myriad.api.model.GetSchedulerStateResponse
> >
> > Feb 22, 2016 6:21:42 PM
> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> > attachTypes
> >
> > INFO: Couldn't find JAX-B element for class
> > org.apache.myriad.api.model.FlexUpServiceRequest
> >
> > Feb 22, 2016 6:21:42 PM
> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> > resolve
> >
> > SEVERE: null
> >
> > E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on fd=231:
> > Transport endpoint is not connected [107]
> >
> > E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on fd=231:
> > Transport endpoint is not connected [107]
> >
> > E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on fd=231:
> > Transport endpoint is not connected [107]
> >
> > *This is the resource manager logs*
> >
> > 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server
> > listener on 8033: starting
> >
> > 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource:
> > Received flexup request. Profile: medium, Instances: 1, Constraints: null
> >
> > 2016-02-22 17:39:31,266 INFO
> org.apache.myriad.scheduler.MyriadOperations:
> > Adding 1 NM instances to cluster
> >
> > 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState:
> Marked
> > taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size of
> > pending queue for nm is: 1
> >
> > 2016-02-22 17:47:25,450 INFO
> >
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
> > Release request cache is cleaned up
> >
> > 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource:
> > Received flexup request. Profile: medium, Instances: 1, Constraints: null
> >
> > 2016-02-22 18:22:22,094 INFO
> org.apache.myriad.scheduler.MyriadOperations:
> > Adding 1 NM instances to cluster
> >
> > 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState:
> Marked
> > taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size of
> > pending queue for nm is: 2
> >
> >
> >
> > Thanks,
> >
>
>
> --
> Dipl.-Inform. Björn Hagemeier
> Federated Systems and Data
> Juelich Supercomputing Centre
> Institute for Advanced Simulation
>
> Phone: +49 2461 61 1584
> Fax : +49 2461 61 6656
> Email: b.hagemeier@fz-juelich.de
> Skype: bhagemeier
> WWW : http://www.fz-juelich.de/jsc
>
> JSC is the coordinator of the
> John von Neumann Institute for Computing
> and member of the
> Gauss Centre for Supercomputing
>
>
> -------------------------------------------------------------------------------------
>
> -------------------------------------------------------------------------------------
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
>
> -------------------------------------------------------------------------------------
>
> -------------------------------------------------------------------------------------
>
>
Re: Not able to launch Node Managers via Myriad
Posted by Björn Hagemeier <b....@fz-juelich.de>.
Hi Mayank,
it seems you and I are having similar issues. Let's see whether they can
be traced to the same causes.
Best regards,
Björn
Am 22.02.2016 um 19:33 schrieb Mayank Bansal:
> Hi,
>
> I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x
>
>
> I have followed all the instructions in the wiki
>
> So the resource manager is up and i am able to launch the web UI of myriad
> as well however whenever I try to use flex up it launches some tasks and
> put them in to pending queue and after that nothing launches.
>
> I am attaching below the resourcemanager.out and log snippets.
>
> Please let me know if I am missing something.
>
> Thanks,
>
>
> *I see some exceptions in resourcemanager.out files those are as follows*
>
>
> Feb 22, 2016 6:21:42 PM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> resolve
>
> SEVERE: null
>
> java.lang.IllegalAccessException: Class
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> can not access a member of class javax.ws.rs.core.Response with modifiers
> "protected"
>
> at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
>
> at java.lang.Class.newInstance(Class.java:436)
>
> at
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
>
> at
> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
>
> at
> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
>
> at
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
>
> at
> com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
>
> at
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
>
> at
> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>
> at
> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>
> at org.mortbay.jetty.Server.handle(Server.java:326)
>
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>
>
> Feb 22, 2016 6:21:42 PM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> attachTypes
>
> INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
>
> Feb 22, 2016 6:21:42 PM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> attachTypes
>
> INFO: Couldn't find JAX-B element for class
> org.apache.myriad.configuration.MyriadConfiguration
>
> Feb 22, 2016 6:21:42 PM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> attachTypes
>
> INFO: Couldn't find JAX-B element for class
> org.apache.myriad.api.model.GetSchedulerStateResponse
>
> Feb 22, 2016 6:21:42 PM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> attachTypes
>
> INFO: Couldn't find JAX-B element for class
> org.apache.myriad.api.model.FlexUpServiceRequest
>
> Feb 22, 2016 6:21:42 PM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> resolve
>
> SEVERE: null
>
> E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on fd=231:
> Transport endpoint is not connected [107]
>
> E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on fd=231:
> Transport endpoint is not connected [107]
>
> E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on fd=231:
> Transport endpoint is not connected [107]
>
> *This is the resource manager logs*
>
> 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 8033: starting
>
> 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource:
> Received flexup request. Profile: medium, Instances: 1, Constraints: null
>
> 2016-02-22 17:39:31,266 INFO org.apache.myriad.scheduler.MyriadOperations:
> Adding 1 NM instances to cluster
>
> 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState: Marked
> taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size of
> pending queue for nm is: 1
>
> 2016-02-22 17:47:25,450 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
> Release request cache is cleaned up
>
> 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource:
> Received flexup request. Profile: medium, Instances: 1, Constraints: null
>
> 2016-02-22 18:22:22,094 INFO org.apache.myriad.scheduler.MyriadOperations:
> Adding 1 NM instances to cluster
>
> 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState: Marked
> taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size of
> pending queue for nm is: 2
>
>
>
> Thanks,
>
--
Dipl.-Inform. Björn Hagemeier
Federated Systems and Data
Juelich Supercomputing Centre
Institute for Advanced Simulation
Phone: +49 2461 61 1584
Fax : +49 2461 61 6656
Email: b.hagemeier@fz-juelich.de
Skype: bhagemeier
WWW : http://www.fz-juelich.de/jsc
JSC is the coordinator of the
John von Neumann Institute for Computing
and member of the
Gauss Centre for Supercomputing
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------