You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon" <ed...@apache.org> on 2014/02/06 12:07:48 UTC
Re: Groomservers IPC server port range
I'll fix soon.
I'm reopen https://issues.apache.org/jira/browse/HAMA-789
On Thu, Feb 6, 2014 at 7:31 PM, Ηλίας Καπουράνης <ik...@csd.auth.gr> wrote:
> Another error came up! Now it starts zookeepers but loses the connection.
> Specifically:
>
> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO ipc.Server:
> Starting Socket Reader #1 for port 49564
> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO ipc.Server: IPC
> Server Responder: starting
> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO ipc.Server: IPC
> Server listener on 49564: starting
> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO
> message.HadoopMessageManagerImpl: BSPPeer address:host port:49564
> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO ipc.Server: IPC
> Server handler 0 on 49564: starting
> attempt_201402061212_0001_000004_0: 14/02/06 12:12:59 INFO
> sync.ZKSyncClient: Initializing ZK Sync Client
> attempt_201402061212_0001_000004_0: 14/02/06 12:12:59 INFO
> sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At
> host/address:49564
> attempt_201402061212_0001_000004_0: 14/02/06 12:12:59 ERROR
> sync.ZooKeeperSyncClientImpl:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /bsp/job_201402061212_0001/peers
> attempt_201402061212_0001_000004_0: 14/02/06 12:13:02 ERROR
> sync.ZKSyncClient: Error checking zk path
> /bsp/job_201402061212_0001/peers/host:49564
> attempt_201402061212_0001_000004_0:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for
> /bsp/job_201402061212_0001/peers/host:49564
>
>
> So I think that the problem again is the port restriction. The 49564 port is
> randomly selected again. I tried seting it at the zookeeper quorum servers
> like host:port1:port2 but still it chooses randomly. Is there any way to
> "standarize" it like with groomserver?
>
>
>
>
> Στις 6/2/2014 10:24 πμ, ο/η Ηλίας Καπουράνης έγραψε:
>
>>
>> Thanks a lot Edward! I will try it and let you know if something comes up!
>>
>> Have a nice day!
>>
>>
>> Στις 6/2/2014 7:15 πμ, ο/η Edward J. Yoon έγραψε:
>>>
>>> Again,
>>>
>>> You can specify the report server port by setting
>>> "bsp.groom.report.address" property like below:
>>>
>>> <property>
>>> <name>bsp.groom.report.address</name>
>>> <value>127.0.0.1:5001</value>
>>> </property>
>>>
>>> Default value was 127.0.0.1:0. (random port).
>>>
>>>
>>>
>>> On Thu, Feb 6, 2014 at 2:09 PM, Edward J. Yoon <ed...@apache.org>
>>> wrote:
>>>>
>>>> Oh, .. thanks.
>>>>
>>>> The GroomServer.taskReportServer seems uses random port. I'll make it
>>>> configurable.
>>>>
>>>> On Wed, Feb 5, 2014 at 10:13 PM, Ηλίας Καπουράνης <ik...@csd.auth.gr>
>>>> wrote:
>>>>>
>>>>> Hey,
>>>>>
>>>>> So what the IPC Server stands for and it gets random ports? I ve seen
>>>>> 39865
>>>>> and 52765. What is the deal with it?
>>>>>
>>>>>
>>>>> Στις 5/2/2014 2:53 μμ, ο/η Edward J. Yoon έγραψε:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> a Hama cluster uses ports "bsp.master.port" 40000 and
>>>>>> "bsp.groom.rpc.port" 50000 to communicate among the cluster nodes.
>>>>>>
>>>>>> And the default peer port is 61000. If you set 3 for the maximum
>>>>>> number of tasks per node, each task uses ports from 61000 to 61003 to
>>>>>> transfer messages between bsp tasks.
>>>>>>
>>>>>> On Wed, Feb 5, 2014 at 9:11 PM, Ηλίας Καπουράνης
>>>>>> <ik...@csd.auth.gr>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hello there,
>>>>>>>
>>>>>>> I am trying to setup Hama in a cluster with really strict port
>>>>>>> permissions
>>>>>>> and I get across the error at every GroomServer:
>>>>>>> "There is a problem in establishing link with BSPMaster."
>>>>>>>
>>>>>>> - Do you know what is the range of ports that every GroomServers IPC
>>>>>>> Server
>>>>>>> gets when started?
>>>>>>> - Is there a way to limit this range to new user values?
>>>>>>>
>>>>>>> Thank you.
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards, Edward J. Yoon
>>>> @eddieyoon
>>>
>>>
>>>
>>
>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Groomservers IPC server port range
Posted by "Edward J. Yoon" <ed...@apache.org>.
Set the bsp.peer.port property.
On Sat, Feb 15, 2014 at 4:48 AM, Ηλίας Καπουράνης <ik...@csd.auth.gr> wrote:
> Hello there,
>
> I've seen that you resolved the HAMA-789. So, is it going to be available
> in a new version? How do we define the specific port we want?
>
>
>
> Στις 6/2/2014 1:07 μμ, ο/η Edward J. Yoon έγραψε:
>
>> I'll fix soon.
>>
>> I'm reopen https://issues.apache.org/jira/browse/HAMA-789
>>
>> On Thu, Feb 6, 2014 at 7:31 PM, Ηλίας Καπουράνης <ik...@csd.auth.gr>
>> wrote:
>>>
>>> Another error came up! Now it starts zookeepers but loses the connection.
>>> Specifically:
>>>
>>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO ipc.Server:
>>> Starting Socket Reader #1 for port 49564
>>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO ipc.Server:
>>> IPC
>>> Server Responder: starting
>>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO ipc.Server:
>>> IPC
>>> Server listener on 49564: starting
>>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO
>>> message.HadoopMessageManagerImpl: BSPPeer address:host port:49564
>>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO ipc.Server:
>>> IPC
>>> Server handler 0 on 49564: starting
>>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:59 INFO
>>> sync.ZKSyncClient: Initializing ZK Sync Client
>>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:59 INFO
>>> sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At
>>> host/address:49564
>>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:59 ERROR
>>> sync.ZooKeeperSyncClientImpl:
>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>> KeeperErrorCode = ConnectionLoss for /bsp/job_201402061212_0001/peers
>>> attempt_201402061212_0001_000004_0: 14/02/06 12:13:02 ERROR
>>> sync.ZKSyncClient: Error checking zk path
>>> /bsp/job_201402061212_0001/peers/host:49564
>>> attempt_201402061212_0001_000004_0:
>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>> KeeperErrorCode = ConnectionLoss for
>>> /bsp/job_201402061212_0001/peers/host:49564
>>>
>>>
>>> So I think that the problem again is the port restriction. The 49564 port
>>> is
>>> randomly selected again. I tried seting it at the zookeeper quorum
>>> servers
>>> like host:port1:port2 but still it chooses randomly. Is there any way to
>>> "standarize" it like with groomserver?
>>>
>>>
>>>
>>>
>>> Στις 6/2/2014 10:24 πμ, ο/η Ηλίας Καπουράνης έγραψε:
>>>
>>>> Thanks a lot Edward! I will try it and let you know if something comes
>>>> up!
>>>>
>>>> Have a nice day!
>>>>
>>>>
>>>> Στις 6/2/2014 7:15 πμ, ο/η Edward J. Yoon έγραψε:
>>>>>
>>>>> Again,
>>>>>
>>>>> You can specify the report server port by setting
>>>>> "bsp.groom.report.address" property like below:
>>>>>
>>>>> <property>
>>>>> <name>bsp.groom.report.address</name>
>>>>> <value>127.0.0.1:5001</value>
>>>>> </property>
>>>>>
>>>>> Default value was 127.0.0.1:0. (random port).
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 2:09 PM, Edward J. Yoon <ed...@apache.org>
>>>>> wrote:
>>>>>>
>>>>>> Oh, .. thanks.
>>>>>>
>>>>>> The GroomServer.taskReportServer seems uses random port. I'll make it
>>>>>> configurable.
>>>>>>
>>>>>> On Wed, Feb 5, 2014 at 10:13 PM, Ηλίας Καπουράνης
>>>>>> <ik...@csd.auth.gr>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hey,
>>>>>>>
>>>>>>> So what the IPC Server stands for and it gets random ports? I ve seen
>>>>>>> 39865
>>>>>>> and 52765. What is the deal with it?
>>>>>>>
>>>>>>>
>>>>>>> Στις 5/2/2014 2:53 μμ, ο/η Edward J. Yoon έγραψε:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> a Hama cluster uses ports "bsp.master.port" 40000 and
>>>>>>>> "bsp.groom.rpc.port" 50000 to communicate among the cluster nodes.
>>>>>>>>
>>>>>>>> And the default peer port is 61000. If you set 3 for the maximum
>>>>>>>> number of tasks per node, each task uses ports from 61000 to 61003
>>>>>>>> to
>>>>>>>> transfer messages between bsp tasks.
>>>>>>>>
>>>>>>>> On Wed, Feb 5, 2014 at 9:11 PM, Ηλίας Καπουράνης
>>>>>>>> <ik...@csd.auth.gr>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hello there,
>>>>>>>>>
>>>>>>>>> I am trying to setup Hama in a cluster with really strict port
>>>>>>>>> permissions
>>>>>>>>> and I get across the error at every GroomServer:
>>>>>>>>> "There is a problem in establishing link with BSPMaster."
>>>>>>>>>
>>>>>>>>> - Do you know what is the range of ports that every GroomServers
>>>>>>>>> IPC
>>>>>>>>> Server
>>>>>>>>> gets when started?
>>>>>>>>> - Is there a way to limit this range to new user values?
>>>>>>>>>
>>>>>>>>> Thank you.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards, Edward J. Yoon
>>>>>> @eddieyoon
>>>>>
>>>>>
>>>>>
>>
>>
>
--
Best Regards, Edward J. Yoon
Chief Executive Officer at DataSayer, Inc.
Re: Groomservers IPC server port range
Posted by Ηλίας Καπουράνης <ik...@csd.auth.gr>.
Hello there,
I've seen that you resolved the HAMA-789. So, is it going to be
available in a new version? How do we define the specific port we want?
Στις 6/2/2014 1:07 μμ, ο/η Edward J. Yoon έγραψε:
> I'll fix soon.
>
> I'm reopen https://issues.apache.org/jira/browse/HAMA-789
>
> On Thu, Feb 6, 2014 at 7:31 PM, Ηλίας Καπουράνης <ik...@csd.auth.gr> wrote:
>> Another error came up! Now it starts zookeepers but loses the connection.
>> Specifically:
>>
>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO ipc.Server:
>> Starting Socket Reader #1 for port 49564
>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO ipc.Server: IPC
>> Server Responder: starting
>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO ipc.Server: IPC
>> Server listener on 49564: starting
>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO
>> message.HadoopMessageManagerImpl: BSPPeer address:host port:49564
>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:58 INFO ipc.Server: IPC
>> Server handler 0 on 49564: starting
>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:59 INFO
>> sync.ZKSyncClient: Initializing ZK Sync Client
>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:59 INFO
>> sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At
>> host/address:49564
>> attempt_201402061212_0001_000004_0: 14/02/06 12:12:59 ERROR
>> sync.ZooKeeperSyncClientImpl:
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /bsp/job_201402061212_0001/peers
>> attempt_201402061212_0001_000004_0: 14/02/06 12:13:02 ERROR
>> sync.ZKSyncClient: Error checking zk path
>> /bsp/job_201402061212_0001/peers/host:49564
>> attempt_201402061212_0001_000004_0:
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for
>> /bsp/job_201402061212_0001/peers/host:49564
>>
>>
>> So I think that the problem again is the port restriction. The 49564 port is
>> randomly selected again. I tried seting it at the zookeeper quorum servers
>> like host:port1:port2 but still it chooses randomly. Is there any way to
>> "standarize" it like with groomserver?
>>
>>
>>
>>
>> Στις 6/2/2014 10:24 πμ, ο/η Ηλίας Καπουράνης έγραψε:
>>
>>> Thanks a lot Edward! I will try it and let you know if something comes up!
>>>
>>> Have a nice day!
>>>
>>>
>>> Στις 6/2/2014 7:15 πμ, ο/η Edward J. Yoon έγραψε:
>>>> Again,
>>>>
>>>> You can specify the report server port by setting
>>>> "bsp.groom.report.address" property like below:
>>>>
>>>> <property>
>>>> <name>bsp.groom.report.address</name>
>>>> <value>127.0.0.1:5001</value>
>>>> </property>
>>>>
>>>> Default value was 127.0.0.1:0. (random port).
>>>>
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 2:09 PM, Edward J. Yoon <ed...@apache.org>
>>>> wrote:
>>>>> Oh, .. thanks.
>>>>>
>>>>> The GroomServer.taskReportServer seems uses random port. I'll make it
>>>>> configurable.
>>>>>
>>>>> On Wed, Feb 5, 2014 at 10:13 PM, Ηλίας Καπουράνης <ik...@csd.auth.gr>
>>>>> wrote:
>>>>>> Hey,
>>>>>>
>>>>>> So what the IPC Server stands for and it gets random ports? I ve seen
>>>>>> 39865
>>>>>> and 52765. What is the deal with it?
>>>>>>
>>>>>>
>>>>>> Στις 5/2/2014 2:53 μμ, ο/η Edward J. Yoon έγραψε:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> a Hama cluster uses ports "bsp.master.port" 40000 and
>>>>>>> "bsp.groom.rpc.port" 50000 to communicate among the cluster nodes.
>>>>>>>
>>>>>>> And the default peer port is 61000. If you set 3 for the maximum
>>>>>>> number of tasks per node, each task uses ports from 61000 to 61003 to
>>>>>>> transfer messages between bsp tasks.
>>>>>>>
>>>>>>> On Wed, Feb 5, 2014 at 9:11 PM, Ηλίας Καπουράνης
>>>>>>> <ik...@csd.auth.gr>
>>>>>>> wrote:
>>>>>>>> Hello there,
>>>>>>>>
>>>>>>>> I am trying to setup Hama in a cluster with really strict port
>>>>>>>> permissions
>>>>>>>> and I get across the error at every GroomServer:
>>>>>>>> "There is a problem in establishing link with BSPMaster."
>>>>>>>>
>>>>>>>> - Do you know what is the range of ports that every GroomServers IPC
>>>>>>>> Server
>>>>>>>> gets when started?
>>>>>>>> - Is there a way to limit this range to new user values?
>>>>>>>>
>>>>>>>> Thank you.
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Best Regards, Edward J. Yoon
>>>>> @eddieyoon
>>>>
>>>>
>
>