You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by Manoj Samel <ma...@gmail.com> on 2016/05/16 20:02:35 UTC
What does PER_CONTAINER in port allocation does ? Facing AM start
error ...
Hello,
When using ALLOCATED_PORT clause, there is a option "PER_CONTAINER".
Can someone explain what does "PER_CONTAINER" option does ? It says keep
port allocation private to container. What does that means ? If multiple
containers are chosen to on same host machine, will this cause issue ?
When using a specific port range using site.global.slider.allowed.ports, I
am getting frequent errors in starting slider AM. The log says
*2016-05-14 20:39:29,236 [main] ERROR main.ServiceLauncher - No available
ports found in configured range {}*
2016-05-14 20:39:29,237 [main] INFO util.ExitUtil - Exiting with status 77
The error is not hard, sometimes same port range works file. I am wondering
if this has anything to do with PER_CONTAINER ...
This is on slider 0.80 on Hadoop 2.6 secured cluster ...
Than
Re: What does PER_CONTAINER in port allocation does ? Facing AM start
error ...
Posted by Manoj Samel <ma...@gmail.com>.
Here is slider.log for Slider AM. Note the port range does not always fails
- it does work occassionally !
"appConf" :{
"schema" : "http://example.org/specification/v2.0.0",
"metadata" : { },
"global" : {
"site.global.qaprod___gms.tenant" : "qaprod___gms",
"site.global.memory_val" : "200m",
"site.global.service_mgmt_log_dir" : "/data/processlauncher/log",
"site.global.xms_val" : "128m",
"site.global.slider.allowed.ports" : "32201–33100",
"java_home" : "/usr/java/latest",
"site.global.xmx_val" : "256m",
"zookeeper.quorum" : "host1:2181,host2:2181,host3:2181",
"site.global.qaprod___gms.principal" : "qaprod.gms",
"env.MALLOC_ARENA_MAX" : "4",
"site.global.qaprod___gms.http_port" :
"${qaprod___gms.ALLOCATED_PORT}{PER_CONTAINER}",
"site.global.bas_host" : "host4",
"site.global.app_version" : "1.0.0",
"site.global.bas_port" : "9009",
"application.def" : ".slider/package/spas/spas-1.0.0.zip",
"site.global.qaprod___gms.keytabfile" :
"/etc/hadoop/conf/qaprod.gms.keytab",
"zookeeper.hosts" : "host1,host2,host3",
"zookeeper.path" : "/services/slider/users/slideradmin/spas",
"site.global.additional_cp" : "/usr/lib/hadoop/lib/*",
"site.global.qaprod___gms.https_port" :
"${qaprod___gms.ALLOCATED_PORT}{PER_CONTAINER}",
"site.fs.defaultFS" : "hdfs://bdsqaprod2",
"site.dfs.namenode.kerberos.principal" : "hdfs/_HOST@BIGDATA",
"site.fs.default.name" : "hdfs://bdsqaprod2"
},
"credentials" : { },
"components" : {
"slider-appmaster" : {
"jvm.heapsize" : "256M",
"slider.keytab.principal.name" : "slideradmin",
"slider.am.keytab.local.path" : "/etc/hadoop/conf/slideradmin.keytab"
},
"qaprod___gms" : { }
}
}}:
2016-05-16 19:24:51,456 [main] INFO appmaster.SliderAppMaster - Conf dir
/hadoop/disk4/yarn/local/usercache/slideradmin/appcache/application_1462834537586_7708/container_e15_1462834537586_7708_09_000001/propagatedconf
does not exist.
2016-05-16 19:24:51,456 [main] INFO appmaster.SliderAppMaster - Parent dir
/hadoop/disk4/yarn/local/usercache/slideradmin/appcache/application_1462834537586_7708/container_e15_1462834537586_7708_09_000001:
tmp
launch_container.sh
container_tokens
lib
confdir
expandedarchive
2016-05-16 19:24:51,457 [main] INFO appmaster.SliderAppMaster - Cluster
provider type is agent
2016-05-16 19:24:51,500 [main] INFO appmaster.SliderAppMaster - RM is at
host9/10.52.88.218:23130
2016-05-16 19:24:51,553 [main] INFO appmaster.SliderAppMaster - AM for ID
7708
2016-05-16 19:24:51,596 [main] INFO impl.NMClientAsyncImpl - Upper bound
of the thread pool size is 500
2016-05-16 19:24:51,597 [main] INFO impl.ContainerManagementProtocolProxy
- yarn.client.max-cached-nodemanagers-proxies : 0
2016-05-16 19:24:51,654 [main] ERROR main.ServiceLauncher - No available
ports found in configured range {}
2016-05-16 19:24:51,656 [main] INFO util.ExitUtil - Exiting with status 77
2016-05-16 19:24:51,658 [Thread-1] INFO appmaster.SliderAppMaster -
Process has exited with exit code 0 mapped to 0 -ignoring
2016-05-16 19:24:51,659 [AMRM Callback Handler Thread] INFO
impl.AMRMClientAsyncImpl - Interrupted while waiting for queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
On Mon, May 16, 2016 at 6:07 PM, Billie Rinaldi <bi...@gmail.com>
wrote:
> Manoj,
>
> Attachments don't come through on ASF mailing lists. What value are you
> using for site.global.slider.allowed.ports?
>
> Billie
>
> On Mon, May 16, 2016 at 4:20 PM, Manoj Samel <ma...@gmail.com>
> wrote:
>
> > Hi Tim,
> >
> > Thanks for your reply
> >
> > In my case, the slider AM itself fails to start. The app remains in
> > accepted state and the container_001 log (the slider AM) errors. I have
> > attached the container_001 slider.log below. It has all the configs,
> > including app config (host names and some other info is altered)
> >
> > The specified port range is not being used on cluster. Also, the error is
> > intermittent i.e. it works for a port range some times and does not works
> > for same for other times. It I removed the allocated_port, it works fine.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, May 16, 2016 at 2:41 PM, Tim I <ti...@timisrael.com> wrote:
> >
> >> I'm pretty sure the PER_CONTAINER statement works as follows for a each
> >> instantiation of a particular service within a slider app:
> >> 1) check the allowable port range
> >> 2) look for an available port on the machine within the allowed port
> range
> >> 3) launch with selected port
> >>
> >> In the case of Accumulo, if you're running two tservers on the same box
> >> and
> >> have a range of ports that they can use, they will not conflict.
> However,
> >> if you statically set the tserver's port and two launch on the same
> >> machine, they will both try to use it and conflict. One will eventually
> >> relaunch on a node with an available port provided resources exist and
> you
> >> don't exceed your failure threshold.
> >>
> >> If you are having an issue with your app master not launching, it might
> be
> >> because you specified a port or port range that is fully utilized by
> >> another application on the same box.
> >>
> >> Is your cluster idle or do you have other slider apps running? Do you
> >> have
> >> more complete output of the logs and possibly the appConfig that you can
> >> share? Are you sure it's the AM failing to start and not a service
> within
> >> your slider app?
> >>
> >> Tim
> >> On May 16, 2016 4:02 PM, "Manoj Samel" <ma...@gmail.com>
> wrote:
> >>
> >> Hello,
> >>
> >> When using ALLOCATED_PORT clause, there is a option "PER_CONTAINER".
> >>
> >> Can someone explain what does "PER_CONTAINER" option does ? It says keep
> >> port allocation private to container. What does that means ? If multiple
> >> containers are chosen to on same host machine, will this cause issue ?
> >>
> >> When using a specific port range using
> site.global.slider.allowed.ports, I
> >> am getting frequent errors in starting slider AM. The log says
> >> *2016-05-14 20:39:29,236 [main] ERROR main.ServiceLauncher - No
> available
> >> ports found in configured range {}*
> >> 2016-05-14 20:39:29,237 [main] INFO util.ExitUtil - Exiting with status
> >> 77
> >>
> >> The error is not hard, sometimes same port range works file. I am
> >> wondering
> >> if this has anything to do with PER_CONTAINER ...
> >>
> >> This is on slider 0.80 on Hadoop 2.6 secured cluster ...
> >>
> >> Than
> >>
> >
> >
>
Re: What does PER_CONTAINER in port allocation does ? Facing AM start
error ...
Posted by Billie Rinaldi <bi...@gmail.com>.
Manoj,
Attachments don't come through on ASF mailing lists. What value are you
using for site.global.slider.allowed.ports?
Billie
On Mon, May 16, 2016 at 4:20 PM, Manoj Samel <ma...@gmail.com>
wrote:
> Hi Tim,
>
> Thanks for your reply
>
> In my case, the slider AM itself fails to start. The app remains in
> accepted state and the container_001 log (the slider AM) errors. I have
> attached the container_001 slider.log below. It has all the configs,
> including app config (host names and some other info is altered)
>
> The specified port range is not being used on cluster. Also, the error is
> intermittent i.e. it works for a port range some times and does not works
> for same for other times. It I removed the allocated_port, it works fine.
>
>
>
>
>
>
>
>
>
> On Mon, May 16, 2016 at 2:41 PM, Tim I <ti...@timisrael.com> wrote:
>
>> I'm pretty sure the PER_CONTAINER statement works as follows for a each
>> instantiation of a particular service within a slider app:
>> 1) check the allowable port range
>> 2) look for an available port on the machine within the allowed port range
>> 3) launch with selected port
>>
>> In the case of Accumulo, if you're running two tservers on the same box
>> and
>> have a range of ports that they can use, they will not conflict. However,
>> if you statically set the tserver's port and two launch on the same
>> machine, they will both try to use it and conflict. One will eventually
>> relaunch on a node with an available port provided resources exist and you
>> don't exceed your failure threshold.
>>
>> If you are having an issue with your app master not launching, it might be
>> because you specified a port or port range that is fully utilized by
>> another application on the same box.
>>
>> Is your cluster idle or do you have other slider apps running? Do you
>> have
>> more complete output of the logs and possibly the appConfig that you can
>> share? Are you sure it's the AM failing to start and not a service within
>> your slider app?
>>
>> Tim
>> On May 16, 2016 4:02 PM, "Manoj Samel" <ma...@gmail.com> wrote:
>>
>> Hello,
>>
>> When using ALLOCATED_PORT clause, there is a option "PER_CONTAINER".
>>
>> Can someone explain what does "PER_CONTAINER" option does ? It says keep
>> port allocation private to container. What does that means ? If multiple
>> containers are chosen to on same host machine, will this cause issue ?
>>
>> When using a specific port range using site.global.slider.allowed.ports, I
>> am getting frequent errors in starting slider AM. The log says
>> *2016-05-14 20:39:29,236 [main] ERROR main.ServiceLauncher - No available
>> ports found in configured range {}*
>> 2016-05-14 20:39:29,237 [main] INFO util.ExitUtil - Exiting with status
>> 77
>>
>> The error is not hard, sometimes same port range works file. I am
>> wondering
>> if this has anything to do with PER_CONTAINER ...
>>
>> This is on slider 0.80 on Hadoop 2.6 secured cluster ...
>>
>> Than
>>
>
>
Re: What does PER_CONTAINER in port allocation does ? Facing AM start
error ...
Posted by Manoj Samel <ma...@gmail.com>.
Hi Tim,
Thanks for your reply
In my case, the slider AM itself fails to start. The app remains in
accepted state and the container_001 log (the slider AM) errors. I have
attached the container_001 slider.log below. It has all the configs,
including app config (host names and some other info is altered)
The specified port range is not being used on cluster. Also, the error is
intermittent i.e. it works for a port range some times and does not works
for same for other times. It I removed the allocated_port, it works fine.
On Mon, May 16, 2016 at 2:41 PM, Tim I <ti...@timisrael.com> wrote:
> I'm pretty sure the PER_CONTAINER statement works as follows for a each
> instantiation of a particular service within a slider app:
> 1) check the allowable port range
> 2) look for an available port on the machine within the allowed port range
> 3) launch with selected port
>
> In the case of Accumulo, if you're running two tservers on the same box and
> have a range of ports that they can use, they will not conflict. However,
> if you statically set the tserver's port and two launch on the same
> machine, they will both try to use it and conflict. One will eventually
> relaunch on a node with an available port provided resources exist and you
> don't exceed your failure threshold.
>
> If you are having an issue with your app master not launching, it might be
> because you specified a port or port range that is fully utilized by
> another application on the same box.
>
> Is your cluster idle or do you have other slider apps running? Do you have
> more complete output of the logs and possibly the appConfig that you can
> share? Are you sure it's the AM failing to start and not a service within
> your slider app?
>
> Tim
> On May 16, 2016 4:02 PM, "Manoj Samel" <ma...@gmail.com> wrote:
>
> Hello,
>
> When using ALLOCATED_PORT clause, there is a option "PER_CONTAINER".
>
> Can someone explain what does "PER_CONTAINER" option does ? It says keep
> port allocation private to container. What does that means ? If multiple
> containers are chosen to on same host machine, will this cause issue ?
>
> When using a specific port range using site.global.slider.allowed.ports, I
> am getting frequent errors in starting slider AM. The log says
> *2016-05-14 20:39:29,236 [main] ERROR main.ServiceLauncher - No available
> ports found in configured range {}*
> 2016-05-14 20:39:29,237 [main] INFO util.ExitUtil - Exiting with status 77
>
> The error is not hard, sometimes same port range works file. I am wondering
> if this has anything to do with PER_CONTAINER ...
>
> This is on slider 0.80 on Hadoop 2.6 secured cluster ...
>
> Than
>
Re: What does PER_CONTAINER in port allocation does ? Facing AM start
error ...
Posted by Tim I <ti...@timisrael.com>.
I'm pretty sure the PER_CONTAINER statement works as follows for a each
instantiation of a particular service within a slider app:
1) check the allowable port range
2) look for an available port on the machine within the allowed port range
3) launch with selected port
In the case of Accumulo, if you're running two tservers on the same box and
have a range of ports that they can use, they will not conflict. However,
if you statically set the tserver's port and two launch on the same
machine, they will both try to use it and conflict. One will eventually
relaunch on a node with an available port provided resources exist and you
don't exceed your failure threshold.
If you are having an issue with your app master not launching, it might be
because you specified a port or port range that is fully utilized by
another application on the same box.
Is your cluster idle or do you have other slider apps running? Do you have
more complete output of the logs and possibly the appConfig that you can
share? Are you sure it's the AM failing to start and not a service within
your slider app?
Tim
On May 16, 2016 4:02 PM, "Manoj Samel" <ma...@gmail.com> wrote:
Hello,
When using ALLOCATED_PORT clause, there is a option "PER_CONTAINER".
Can someone explain what does "PER_CONTAINER" option does ? It says keep
port allocation private to container. What does that means ? If multiple
containers are chosen to on same host machine, will this cause issue ?
When using a specific port range using site.global.slider.allowed.ports, I
am getting frequent errors in starting slider AM. The log says
*2016-05-14 20:39:29,236 [main] ERROR main.ServiceLauncher - No available
ports found in configured range {}*
2016-05-14 20:39:29,237 [main] INFO util.ExitUtil - Exiting with status 77
The error is not hard, sometimes same port range works file. I am wondering
if this has anything to do with PER_CONTAINER ...
This is on slider 0.80 on Hadoop 2.6 secured cluster ...
Than