You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Wido den Hollander <wi...@widodh.nl> on 2013/04/02 20:32:40 UTC
[ACS41] Management server closes connection on port 8250
Hi,
Since I've upgraded my management server to 4.1 my agents refuse to
connect, their log says:
2013-04-02 20:26:11,207 INFO [utils.nio.NioClient]
(Agent-Selector:null) Connecting to 31.25.X.X:8250
2013-04-02 20:26:11,209 ERROR [utils.nio.NioConnection]
(Agent-Selector:null) Unable to initialize the threads.
java.io.IOException: SSL: Fail to init SSL! java.io.IOException:
Connection closed with -1 on reading size.
at com.cloud.utils.nio.NioClient.init(NioClient.java:83)
at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
at java.lang.Thread.run(Thread.java:679)
So I tried a telnet connection:
Connection closed by foreign host.
root@stack01:~# telnet 31.25.X.X 8250
Trying 31.25.X.X...
Connected to 31.X.X.X.
Escape character is '^]'.
Connection closed by foreign host.
root@stack01:~#
So I didn't close the connection, but it was the management server.
I cranked up the logging to DEBUG, but nothing shows in any of the logs,
so I have no clue why this isn't working.
On the mgmt server I see Java in LISTEN state on port 8250
There is no firewall on the management server (it's my lab!).
Any clues to what this could be before I start filing a in Jira? Since
I'm not sure if this is a bug.
Wido
Re: [ACS41] Management server closes connection on port 8250
Posted by Kelven Yang <ke...@citrix.com>.
From: Sheng Yang <sh...@yasker.org>>
Date: Wednesday, April 3, 2013 10:44 AM
To: "<de...@cloudstack.apache.org>>" <de...@cloudstack.apache.org>>, Kelven Yang <ke...@citrix.com>>
Subject: Re: [ACS41] Management server closes connection on port 8250
On Tue, Apr 2, 2013 at 1:12 PM, Chip Childers <ch...@sungard.com>> wrote:
On Tue, Apr 02, 2013 at 10:10:28PM +0200, Wido den Hollander wrote:
> Hi,
>
> On 04/02/2013 08:50 PM, Sheng Yang wrote:
> >On Tue, Apr 2, 2013 at 11:49 AM, Sheng Yang <sh...@yasker.org>> wrote:
> >>On Tue, Apr 2, 2013 at 11:32 AM, Wido den Hollander <wi...@widodh.nl>> wrote:
> >>>Hi,
> >>>
> >>>Since I've upgraded my management server to 4.1 my agents refuse to connect,
> >>>their log says:
> >>>
> >>>2013-04-02 20:26:11,207 INFO [utils.nio.NioClient] (Agent-Selector:null)
> >>>Connecting to 31.25.X.X:8250
> >>>2013-04-02 20:26:11,209 ERROR [utils.nio.NioConnection]
> >>>(Agent-Selector:null) Unable to initialize the threads.
> >>>java.io.IOException: SSL: Fail to init SSL! java.io.IOException: Connection
> >>>closed with -1 on reading size.
> >>> at com.cloud.utils.nio.NioClient.init(NioClient.java:83)
> >>> at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
> >>> at java.lang.Thread.run(Thread.java:679)
> >>>
> >>>So I tried a telnet connection:
> >>>
> >>>Connection closed by foreign host.
> >>>root@stack01:~# telnet 31.25.X.X 8250
> >>>Trying 31.25.X.X...
> >>>Connected to 31.X.X.X.
> >>>Escape character is '^]'.
> >>>Connection closed by foreign host.
> >>>root@stack01:~#
> >>>
> >>>So I didn't close the connection, but it was the management server.
> >>>
> >>>I cranked up the logging to DEBUG, but nothing shows in any of the logs, so
> >>>I have no clue why this isn't working.
> >>>
> >>>On the mgmt server I see Java in LISTEN state on port 8250
> >>>
> >>>There is no firewall on the management server (it's my lab!).
> >>>
> >>>Any clues to what this could be before I start filing a in Jira? Since I'm
> >>>not sure if this is a bug.
> >>
> >>Have you upgrade your agent?
> >
>
> Yes, the Agent is also 4.1
>
> >Also you could try to enable TRACE for com.cloud.utils.nio to see more
> >log of NIO.
> >
>
> I think you mean at the agent?
>
> But what I think is weird is that the mgmt server directly closes
> the telnet connection as well and nothing shows up in the logs.
>
> In the mgmt server log I found this:
>
> 2013-04-02 22:08:41,156 INFO [utils.nio.NioServer]
> (AgentManager-Selector:null) NioConnection started and listening on
> 0.0.0.0/0.0.0.0:8250<http://0.0.0.0/0.0.0.0:8250>
>
> What I did notice is that when I shutdown the mgmt server the Agent
> connects for a second:
>
> 2013-04-02 22:07:53,689 INFO [utils.nio.NioClient]
> (Agent-Selector:null) Connecting to 31.25.X.X:8250
> 2013-04-02 22:07:53,934 INFO [utils.nio.NioClient]
> (Agent-Selector:null) SSL: Handshake done
>
> And then the mgmt server exists and the connection gets lost.
>
> ANY clues about this weird behavior?
>
> Wido
>
> >--Sheng
> >>
> >>--Sheng
> >>>
> >>>Wido
>
Wido - Can you please open a blocker bug to track this? It may end up
being a non-issue or something... but I want to know that we have it
outstanding when considering when to cut the RC.
It remind me of one weird bug. The corrupted ssl keystore(due to some encryption/decryption of db which I don't understand well, I remember Kelven knows more about it) on mgmt server make it impossible to connect to mgmt server.
Yes, I encountered an issue in upgrade case where password related fields seem to be encrypted/corrupted after the migration process. It it worth to dig it out for a better explanation
-kelven
Could you clear "ssl.keystore" entry in configuration table of db, as well as cloud.keystore file in the configuration path and restart the mgmt server and try again?
--Sheng
Re: [ACS41] Management server closes connection on port 8250
Posted by Wido den Hollander <wi...@widodh.nl>.
On 04/03/2013 08:14 PM, Marcus Sorensen wrote:
> did you copy the key file from the old install?
>
Yes, I copied "key" which only contained "password".
Still not sure about it though. What I did find out, we need to do a
trim on the contents of the key file, since a small whitespace at the
end will already break your setup.
Wido
>
> On Wed, Apr 3, 2013 at 12:06 PM, Wido den Hollander <wi...@widodh.nl> wrote:
>
>>
>>
>> On 04/03/2013 07:44 PM, Sheng Yang wrote:
>>
>>> On Tue, Apr 2, 2013 at 1:12 PM, Chip Childers <ch...@sungard.com>
>>> **wrote:
>>>
>>> On Tue, Apr 02, 2013 at 10:10:28PM +0200, Wido den Hollander wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> On 04/02/2013 08:50 PM, Sheng Yang wrote:
>>>>>
>>>>>> On Tue, Apr 2, 2013 at 11:49 AM, Sheng Yang <sh...@yasker.org> wrote:
>>>>>>
>>>>>>> On Tue, Apr 2, 2013 at 11:32 AM, Wido den Hollander <wi...@widodh.nl>
>>>>>>>
>>>>>> wrote:
>>>>
>>>>> Hi,
>>>>>>>>
>>>>>>>> Since I've upgraded my management server to 4.1 my agents refuse to
>>>>>>>>
>>>>>>> connect,
>>>>
>>>>> their log says:
>>>>>>>>
>>>>>>>> 2013-04-02 20:26:11,207 INFO [utils.nio.NioClient]
>>>>>>>>
>>>>>>> (Agent-Selector:null)
>>>>
>>>>> Connecting to 31.25.X.X:8250
>>>>>>>> 2013-04-02 20:26:11,209 ERROR [utils.nio.NioConnection]
>>>>>>>> (Agent-Selector:null) Unable to initialize the threads.
>>>>>>>> java.io.IOException: SSL: Fail to init SSL! java.io.IOException:
>>>>>>>>
>>>>>>> Connection
>>>>
>>>>> closed with -1 on reading size.
>>>>>>>> at com.cloud.utils.nio.NioClient.**init(NioClient.java:83)
>>>>>>>> at
>>>>>>>>
>>>>>>> com.cloud.utils.nio.**NioConnection.run(**NioConnection.java:108)
>>>>
>>>>> at java.lang.Thread.run(Thread.**java:679)
>>>>>>>>
>>>>>>>> So I tried a telnet connection:
>>>>>>>>
>>>>>>>> Connection closed by foreign host.
>>>>>>>> root@stack01:~# telnet 31.25.X.X 8250
>>>>>>>> Trying 31.25.X.X...
>>>>>>>> Connected to 31.X.X.X.
>>>>>>>> Escape character is '^]'.
>>>>>>>> Connection closed by foreign host.
>>>>>>>> root@stack01:~#
>>>>>>>>
>>>>>>>> So I didn't close the connection, but it was the management server.
>>>>>>>>
>>>>>>>> I cranked up the logging to DEBUG, but nothing shows in any of the
>>>>>>>>
>>>>>>> logs, so
>>>>
>>>>> I have no clue why this isn't working.
>>>>>>>>
>>>>>>>> On the mgmt server I see Java in LISTEN state on port 8250
>>>>>>>>
>>>>>>>> There is no firewall on the management server (it's my lab!).
>>>>>>>>
>>>>>>>> Any clues to what this could be before I start filing a in Jira?
>>>>>>>>
>>>>>>> Since I'm
>>>>
>>>>> not sure if this is a bug.
>>>>>>>>
>>>>>>>
>>>>>>> Have you upgrade your agent?
>>>>>>>
>>>>>>
>>>>>>
>>>>> Yes, the Agent is also 4.1
>>>>>
>>>>> Also you could try to enable TRACE for com.cloud.utils.nio to see more
>>>>>> log of NIO.
>>>>>>
>>>>>>
>>>>> I think you mean at the agent?
>>>>>
>>>>> But what I think is weird is that the mgmt server directly closes
>>>>> the telnet connection as well and nothing shows up in the logs.
>>>>>
>>>>> In the mgmt server log I found this:
>>>>>
>>>>> 2013-04-02 22:08:41,156 INFO [utils.nio.NioServer]
>>>>> (AgentManager-Selector:null) NioConnection started and listening on
>>>>> 0.0.0.0/0.0.0.0:8250
>>>>>
>>>>> What I did notice is that when I shutdown the mgmt server the Agent
>>>>> connects for a second:
>>>>>
>>>>> 2013-04-02 22:07:53,689 INFO [utils.nio.NioClient]
>>>>> (Agent-Selector:null) Connecting to 31.25.X.X:8250
>>>>> 2013-04-02 22:07:53,934 INFO [utils.nio.NioClient]
>>>>> (Agent-Selector:null) SSL: Handshake done
>>>>>
>>>>> And then the mgmt server exists and the connection gets lost.
>>>>>
>>>>> ANY clues about this weird behavior?
>>>>>
>>>>> Wido
>>>>>
>>>>> --Sheng
>>>>>>
>>>>>>>
>>>>>>> --Sheng
>>>>>>>
>>>>>>>>
>>>>>>>> Wido
>>>>>>>>
>>>>>>>
>>>>>
>>>> Wido - Can you please open a blocker bug to track this? It may end up
>>>> being a non-issue or something... but I want to know that we have it
>>>> outstanding when considering when to cut the RC.
>>>>
>>>>
>>> It remind me of one weird bug. The corrupted ssl keystore(due to some
>>> encryption/decryption of db which I don't understand well, I remember
>>> Kelven knows more about it) on mgmt server make it impossible to connect
>>> to
>>> mgmt server.
>>>
>>> Could you clear "ssl.keystore" entry in configuration table of db, as well
>>> as cloud.keystore file in the configuration path and restart the mgmt
>>> server and try again?
>>>
>>>
>> I "fixed" it about 10 minutes prior to your e-mail. It seems related to
>> CLOUDSTACK-1877: https://issues.apache.org/**jira/browse/CLOUDSTACK-1877<https://issues.apache.org/jira/browse/CLOUDSTACK-1877>
>>
>> The management server read the wrong db.properties and after fixing that
>> it worked.
>>
>> It seems to be due to some encryption issues because I had to completely
>> turn of db encryption in my db.properties.
>>
>> My mgmt server is now running, but not sure yet what the root cause is.
>>
>> Wido
>>
>> --Sheng
>>>
>>>
>
Re: [ACS41] Management server closes connection on port 8250
Posted by Marcus Sorensen <sh...@gmail.com>.
did you copy the key file from the old install?
On Wed, Apr 3, 2013 at 12:06 PM, Wido den Hollander <wi...@widodh.nl> wrote:
>
>
> On 04/03/2013 07:44 PM, Sheng Yang wrote:
>
>> On Tue, Apr 2, 2013 at 1:12 PM, Chip Childers <ch...@sungard.com>
>> **wrote:
>>
>> On Tue, Apr 02, 2013 at 10:10:28PM +0200, Wido den Hollander wrote:
>>>
>>>> Hi,
>>>>
>>>> On 04/02/2013 08:50 PM, Sheng Yang wrote:
>>>>
>>>>> On Tue, Apr 2, 2013 at 11:49 AM, Sheng Yang <sh...@yasker.org> wrote:
>>>>>
>>>>>> On Tue, Apr 2, 2013 at 11:32 AM, Wido den Hollander <wi...@widodh.nl>
>>>>>>
>>>>> wrote:
>>>
>>>> Hi,
>>>>>>>
>>>>>>> Since I've upgraded my management server to 4.1 my agents refuse to
>>>>>>>
>>>>>> connect,
>>>
>>>> their log says:
>>>>>>>
>>>>>>> 2013-04-02 20:26:11,207 INFO [utils.nio.NioClient]
>>>>>>>
>>>>>> (Agent-Selector:null)
>>>
>>>> Connecting to 31.25.X.X:8250
>>>>>>> 2013-04-02 20:26:11,209 ERROR [utils.nio.NioConnection]
>>>>>>> (Agent-Selector:null) Unable to initialize the threads.
>>>>>>> java.io.IOException: SSL: Fail to init SSL! java.io.IOException:
>>>>>>>
>>>>>> Connection
>>>
>>>> closed with -1 on reading size.
>>>>>>> at com.cloud.utils.nio.NioClient.**init(NioClient.java:83)
>>>>>>> at
>>>>>>>
>>>>>> com.cloud.utils.nio.**NioConnection.run(**NioConnection.java:108)
>>>
>>>> at java.lang.Thread.run(Thread.**java:679)
>>>>>>>
>>>>>>> So I tried a telnet connection:
>>>>>>>
>>>>>>> Connection closed by foreign host.
>>>>>>> root@stack01:~# telnet 31.25.X.X 8250
>>>>>>> Trying 31.25.X.X...
>>>>>>> Connected to 31.X.X.X.
>>>>>>> Escape character is '^]'.
>>>>>>> Connection closed by foreign host.
>>>>>>> root@stack01:~#
>>>>>>>
>>>>>>> So I didn't close the connection, but it was the management server.
>>>>>>>
>>>>>>> I cranked up the logging to DEBUG, but nothing shows in any of the
>>>>>>>
>>>>>> logs, so
>>>
>>>> I have no clue why this isn't working.
>>>>>>>
>>>>>>> On the mgmt server I see Java in LISTEN state on port 8250
>>>>>>>
>>>>>>> There is no firewall on the management server (it's my lab!).
>>>>>>>
>>>>>>> Any clues to what this could be before I start filing a in Jira?
>>>>>>>
>>>>>> Since I'm
>>>
>>>> not sure if this is a bug.
>>>>>>>
>>>>>>
>>>>>> Have you upgrade your agent?
>>>>>>
>>>>>
>>>>>
>>>> Yes, the Agent is also 4.1
>>>>
>>>> Also you could try to enable TRACE for com.cloud.utils.nio to see more
>>>>> log of NIO.
>>>>>
>>>>>
>>>> I think you mean at the agent?
>>>>
>>>> But what I think is weird is that the mgmt server directly closes
>>>> the telnet connection as well and nothing shows up in the logs.
>>>>
>>>> In the mgmt server log I found this:
>>>>
>>>> 2013-04-02 22:08:41,156 INFO [utils.nio.NioServer]
>>>> (AgentManager-Selector:null) NioConnection started and listening on
>>>> 0.0.0.0/0.0.0.0:8250
>>>>
>>>> What I did notice is that when I shutdown the mgmt server the Agent
>>>> connects for a second:
>>>>
>>>> 2013-04-02 22:07:53,689 INFO [utils.nio.NioClient]
>>>> (Agent-Selector:null) Connecting to 31.25.X.X:8250
>>>> 2013-04-02 22:07:53,934 INFO [utils.nio.NioClient]
>>>> (Agent-Selector:null) SSL: Handshake done
>>>>
>>>> And then the mgmt server exists and the connection gets lost.
>>>>
>>>> ANY clues about this weird behavior?
>>>>
>>>> Wido
>>>>
>>>> --Sheng
>>>>>
>>>>>>
>>>>>> --Sheng
>>>>>>
>>>>>>>
>>>>>>> Wido
>>>>>>>
>>>>>>
>>>>
>>> Wido - Can you please open a blocker bug to track this? It may end up
>>> being a non-issue or something... but I want to know that we have it
>>> outstanding when considering when to cut the RC.
>>>
>>>
>> It remind me of one weird bug. The corrupted ssl keystore(due to some
>> encryption/decryption of db which I don't understand well, I remember
>> Kelven knows more about it) on mgmt server make it impossible to connect
>> to
>> mgmt server.
>>
>> Could you clear "ssl.keystore" entry in configuration table of db, as well
>> as cloud.keystore file in the configuration path and restart the mgmt
>> server and try again?
>>
>>
> I "fixed" it about 10 minutes prior to your e-mail. It seems related to
> CLOUDSTACK-1877: https://issues.apache.org/**jira/browse/CLOUDSTACK-1877<https://issues.apache.org/jira/browse/CLOUDSTACK-1877>
>
> The management server read the wrong db.properties and after fixing that
> it worked.
>
> It seems to be due to some encryption issues because I had to completely
> turn of db encryption in my db.properties.
>
> My mgmt server is now running, but not sure yet what the root cause is.
>
> Wido
>
> --Sheng
>>
>>
Re: [ACS41] Management server closes connection on port 8250
Posted by Wido den Hollander <wi...@widodh.nl>.
On 04/03/2013 07:44 PM, Sheng Yang wrote:
> On Tue, Apr 2, 2013 at 1:12 PM, Chip Childers <ch...@sungard.com>wrote:
>
>> On Tue, Apr 02, 2013 at 10:10:28PM +0200, Wido den Hollander wrote:
>>> Hi,
>>>
>>> On 04/02/2013 08:50 PM, Sheng Yang wrote:
>>>> On Tue, Apr 2, 2013 at 11:49 AM, Sheng Yang <sh...@yasker.org> wrote:
>>>>> On Tue, Apr 2, 2013 at 11:32 AM, Wido den Hollander <wi...@widodh.nl>
>> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Since I've upgraded my management server to 4.1 my agents refuse to
>> connect,
>>>>>> their log says:
>>>>>>
>>>>>> 2013-04-02 20:26:11,207 INFO [utils.nio.NioClient]
>> (Agent-Selector:null)
>>>>>> Connecting to 31.25.X.X:8250
>>>>>> 2013-04-02 20:26:11,209 ERROR [utils.nio.NioConnection]
>>>>>> (Agent-Selector:null) Unable to initialize the threads.
>>>>>> java.io.IOException: SSL: Fail to init SSL! java.io.IOException:
>> Connection
>>>>>> closed with -1 on reading size.
>>>>>> at com.cloud.utils.nio.NioClient.init(NioClient.java:83)
>>>>>> at
>> com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
>>>>>> at java.lang.Thread.run(Thread.java:679)
>>>>>>
>>>>>> So I tried a telnet connection:
>>>>>>
>>>>>> Connection closed by foreign host.
>>>>>> root@stack01:~# telnet 31.25.X.X 8250
>>>>>> Trying 31.25.X.X...
>>>>>> Connected to 31.X.X.X.
>>>>>> Escape character is '^]'.
>>>>>> Connection closed by foreign host.
>>>>>> root@stack01:~#
>>>>>>
>>>>>> So I didn't close the connection, but it was the management server.
>>>>>>
>>>>>> I cranked up the logging to DEBUG, but nothing shows in any of the
>> logs, so
>>>>>> I have no clue why this isn't working.
>>>>>>
>>>>>> On the mgmt server I see Java in LISTEN state on port 8250
>>>>>>
>>>>>> There is no firewall on the management server (it's my lab!).
>>>>>>
>>>>>> Any clues to what this could be before I start filing a in Jira?
>> Since I'm
>>>>>> not sure if this is a bug.
>>>>>
>>>>> Have you upgrade your agent?
>>>>
>>>
>>> Yes, the Agent is also 4.1
>>>
>>>> Also you could try to enable TRACE for com.cloud.utils.nio to see more
>>>> log of NIO.
>>>>
>>>
>>> I think you mean at the agent?
>>>
>>> But what I think is weird is that the mgmt server directly closes
>>> the telnet connection as well and nothing shows up in the logs.
>>>
>>> In the mgmt server log I found this:
>>>
>>> 2013-04-02 22:08:41,156 INFO [utils.nio.NioServer]
>>> (AgentManager-Selector:null) NioConnection started and listening on
>>> 0.0.0.0/0.0.0.0:8250
>>>
>>> What I did notice is that when I shutdown the mgmt server the Agent
>>> connects for a second:
>>>
>>> 2013-04-02 22:07:53,689 INFO [utils.nio.NioClient]
>>> (Agent-Selector:null) Connecting to 31.25.X.X:8250
>>> 2013-04-02 22:07:53,934 INFO [utils.nio.NioClient]
>>> (Agent-Selector:null) SSL: Handshake done
>>>
>>> And then the mgmt server exists and the connection gets lost.
>>>
>>> ANY clues about this weird behavior?
>>>
>>> Wido
>>>
>>>> --Sheng
>>>>>
>>>>> --Sheng
>>>>>>
>>>>>> Wido
>>>
>>
>> Wido - Can you please open a blocker bug to track this? It may end up
>> being a non-issue or something... but I want to know that we have it
>> outstanding when considering when to cut the RC.
>>
>
> It remind me of one weird bug. The corrupted ssl keystore(due to some
> encryption/decryption of db which I don't understand well, I remember
> Kelven knows more about it) on mgmt server make it impossible to connect to
> mgmt server.
>
> Could you clear "ssl.keystore" entry in configuration table of db, as well
> as cloud.keystore file in the configuration path and restart the mgmt
> server and try again?
>
I "fixed" it about 10 minutes prior to your e-mail. It seems related to
CLOUDSTACK-1877: https://issues.apache.org/jira/browse/CLOUDSTACK-1877
The management server read the wrong db.properties and after fixing that
it worked.
It seems to be due to some encryption issues because I had to completely
turn of db encryption in my db.properties.
My mgmt server is now running, but not sure yet what the root cause is.
Wido
> --Sheng
>
Re: [ACS41] Management server closes connection on port 8250
Posted by Sheng Yang <sh...@yasker.org>.
On Tue, Apr 2, 2013 at 1:12 PM, Chip Childers <ch...@sungard.com>wrote:
> On Tue, Apr 02, 2013 at 10:10:28PM +0200, Wido den Hollander wrote:
> > Hi,
> >
> > On 04/02/2013 08:50 PM, Sheng Yang wrote:
> > >On Tue, Apr 2, 2013 at 11:49 AM, Sheng Yang <sh...@yasker.org> wrote:
> > >>On Tue, Apr 2, 2013 at 11:32 AM, Wido den Hollander <wi...@widodh.nl>
> wrote:
> > >>>Hi,
> > >>>
> > >>>Since I've upgraded my management server to 4.1 my agents refuse to
> connect,
> > >>>their log says:
> > >>>
> > >>>2013-04-02 20:26:11,207 INFO [utils.nio.NioClient]
> (Agent-Selector:null)
> > >>>Connecting to 31.25.X.X:8250
> > >>>2013-04-02 20:26:11,209 ERROR [utils.nio.NioConnection]
> > >>>(Agent-Selector:null) Unable to initialize the threads.
> > >>>java.io.IOException: SSL: Fail to init SSL! java.io.IOException:
> Connection
> > >>>closed with -1 on reading size.
> > >>> at com.cloud.utils.nio.NioClient.init(NioClient.java:83)
> > >>> at
> com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
> > >>> at java.lang.Thread.run(Thread.java:679)
> > >>>
> > >>>So I tried a telnet connection:
> > >>>
> > >>>Connection closed by foreign host.
> > >>>root@stack01:~# telnet 31.25.X.X 8250
> > >>>Trying 31.25.X.X...
> > >>>Connected to 31.X.X.X.
> > >>>Escape character is '^]'.
> > >>>Connection closed by foreign host.
> > >>>root@stack01:~#
> > >>>
> > >>>So I didn't close the connection, but it was the management server.
> > >>>
> > >>>I cranked up the logging to DEBUG, but nothing shows in any of the
> logs, so
> > >>>I have no clue why this isn't working.
> > >>>
> > >>>On the mgmt server I see Java in LISTEN state on port 8250
> > >>>
> > >>>There is no firewall on the management server (it's my lab!).
> > >>>
> > >>>Any clues to what this could be before I start filing a in Jira?
> Since I'm
> > >>>not sure if this is a bug.
> > >>
> > >>Have you upgrade your agent?
> > >
> >
> > Yes, the Agent is also 4.1
> >
> > >Also you could try to enable TRACE for com.cloud.utils.nio to see more
> > >log of NIO.
> > >
> >
> > I think you mean at the agent?
> >
> > But what I think is weird is that the mgmt server directly closes
> > the telnet connection as well and nothing shows up in the logs.
> >
> > In the mgmt server log I found this:
> >
> > 2013-04-02 22:08:41,156 INFO [utils.nio.NioServer]
> > (AgentManager-Selector:null) NioConnection started and listening on
> > 0.0.0.0/0.0.0.0:8250
> >
> > What I did notice is that when I shutdown the mgmt server the Agent
> > connects for a second:
> >
> > 2013-04-02 22:07:53,689 INFO [utils.nio.NioClient]
> > (Agent-Selector:null) Connecting to 31.25.X.X:8250
> > 2013-04-02 22:07:53,934 INFO [utils.nio.NioClient]
> > (Agent-Selector:null) SSL: Handshake done
> >
> > And then the mgmt server exists and the connection gets lost.
> >
> > ANY clues about this weird behavior?
> >
> > Wido
> >
> > >--Sheng
> > >>
> > >>--Sheng
> > >>>
> > >>>Wido
> >
>
> Wido - Can you please open a blocker bug to track this? It may end up
> being a non-issue or something... but I want to know that we have it
> outstanding when considering when to cut the RC.
>
It remind me of one weird bug. The corrupted ssl keystore(due to some
encryption/decryption of db which I don't understand well, I remember
Kelven knows more about it) on mgmt server make it impossible to connect to
mgmt server.
Could you clear "ssl.keystore" entry in configuration table of db, as well
as cloud.keystore file in the configuration path and restart the mgmt
server and try again?
--Sheng
Re: [ACS41] Management server closes connection on port 8250
Posted by Chip Childers <ch...@sungard.com>.
On Tue, Apr 02, 2013 at 10:10:28PM +0200, Wido den Hollander wrote:
> Hi,
>
> On 04/02/2013 08:50 PM, Sheng Yang wrote:
> >On Tue, Apr 2, 2013 at 11:49 AM, Sheng Yang <sh...@yasker.org> wrote:
> >>On Tue, Apr 2, 2013 at 11:32 AM, Wido den Hollander <wi...@widodh.nl> wrote:
> >>>Hi,
> >>>
> >>>Since I've upgraded my management server to 4.1 my agents refuse to connect,
> >>>their log says:
> >>>
> >>>2013-04-02 20:26:11,207 INFO [utils.nio.NioClient] (Agent-Selector:null)
> >>>Connecting to 31.25.X.X:8250
> >>>2013-04-02 20:26:11,209 ERROR [utils.nio.NioConnection]
> >>>(Agent-Selector:null) Unable to initialize the threads.
> >>>java.io.IOException: SSL: Fail to init SSL! java.io.IOException: Connection
> >>>closed with -1 on reading size.
> >>> at com.cloud.utils.nio.NioClient.init(NioClient.java:83)
> >>> at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
> >>> at java.lang.Thread.run(Thread.java:679)
> >>>
> >>>So I tried a telnet connection:
> >>>
> >>>Connection closed by foreign host.
> >>>root@stack01:~# telnet 31.25.X.X 8250
> >>>Trying 31.25.X.X...
> >>>Connected to 31.X.X.X.
> >>>Escape character is '^]'.
> >>>Connection closed by foreign host.
> >>>root@stack01:~#
> >>>
> >>>So I didn't close the connection, but it was the management server.
> >>>
> >>>I cranked up the logging to DEBUG, but nothing shows in any of the logs, so
> >>>I have no clue why this isn't working.
> >>>
> >>>On the mgmt server I see Java in LISTEN state on port 8250
> >>>
> >>>There is no firewall on the management server (it's my lab!).
> >>>
> >>>Any clues to what this could be before I start filing a in Jira? Since I'm
> >>>not sure if this is a bug.
> >>
> >>Have you upgrade your agent?
> >
>
> Yes, the Agent is also 4.1
>
> >Also you could try to enable TRACE for com.cloud.utils.nio to see more
> >log of NIO.
> >
>
> I think you mean at the agent?
>
> But what I think is weird is that the mgmt server directly closes
> the telnet connection as well and nothing shows up in the logs.
>
> In the mgmt server log I found this:
>
> 2013-04-02 22:08:41,156 INFO [utils.nio.NioServer]
> (AgentManager-Selector:null) NioConnection started and listening on
> 0.0.0.0/0.0.0.0:8250
>
> What I did notice is that when I shutdown the mgmt server the Agent
> connects for a second:
>
> 2013-04-02 22:07:53,689 INFO [utils.nio.NioClient]
> (Agent-Selector:null) Connecting to 31.25.X.X:8250
> 2013-04-02 22:07:53,934 INFO [utils.nio.NioClient]
> (Agent-Selector:null) SSL: Handshake done
>
> And then the mgmt server exists and the connection gets lost.
>
> ANY clues about this weird behavior?
>
> Wido
>
> >--Sheng
> >>
> >>--Sheng
> >>>
> >>>Wido
>
Wido - Can you please open a blocker bug to track this? It may end up
being a non-issue or something... but I want to know that we have it
outstanding when considering when to cut the RC.
Re: [ACS41] Management server closes connection on port 8250
Posted by Wido den Hollander <wi...@widodh.nl>.
Hi,
On 04/02/2013 08:50 PM, Sheng Yang wrote:
> On Tue, Apr 2, 2013 at 11:49 AM, Sheng Yang <sh...@yasker.org> wrote:
>> On Tue, Apr 2, 2013 at 11:32 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>>> Hi,
>>>
>>> Since I've upgraded my management server to 4.1 my agents refuse to connect,
>>> their log says:
>>>
>>> 2013-04-02 20:26:11,207 INFO [utils.nio.NioClient] (Agent-Selector:null)
>>> Connecting to 31.25.X.X:8250
>>> 2013-04-02 20:26:11,209 ERROR [utils.nio.NioConnection]
>>> (Agent-Selector:null) Unable to initialize the threads.
>>> java.io.IOException: SSL: Fail to init SSL! java.io.IOException: Connection
>>> closed with -1 on reading size.
>>> at com.cloud.utils.nio.NioClient.init(NioClient.java:83)
>>> at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
>>> at java.lang.Thread.run(Thread.java:679)
>>>
>>> So I tried a telnet connection:
>>>
>>> Connection closed by foreign host.
>>> root@stack01:~# telnet 31.25.X.X 8250
>>> Trying 31.25.X.X...
>>> Connected to 31.X.X.X.
>>> Escape character is '^]'.
>>> Connection closed by foreign host.
>>> root@stack01:~#
>>>
>>> So I didn't close the connection, but it was the management server.
>>>
>>> I cranked up the logging to DEBUG, but nothing shows in any of the logs, so
>>> I have no clue why this isn't working.
>>>
>>> On the mgmt server I see Java in LISTEN state on port 8250
>>>
>>> There is no firewall on the management server (it's my lab!).
>>>
>>> Any clues to what this could be before I start filing a in Jira? Since I'm
>>> not sure if this is a bug.
>>
>> Have you upgrade your agent?
>
Yes, the Agent is also 4.1
> Also you could try to enable TRACE for com.cloud.utils.nio to see more
> log of NIO.
>
I think you mean at the agent?
But what I think is weird is that the mgmt server directly closes the
telnet connection as well and nothing shows up in the logs.
In the mgmt server log I found this:
2013-04-02 22:08:41,156 INFO [utils.nio.NioServer]
(AgentManager-Selector:null) NioConnection started and listening on
0.0.0.0/0.0.0.0:8250
What I did notice is that when I shutdown the mgmt server the Agent
connects for a second:
2013-04-02 22:07:53,689 INFO [utils.nio.NioClient]
(Agent-Selector:null) Connecting to 31.25.X.X:8250
2013-04-02 22:07:53,934 INFO [utils.nio.NioClient]
(Agent-Selector:null) SSL: Handshake done
And then the mgmt server exists and the connection gets lost.
ANY clues about this weird behavior?
Wido
> --Sheng
>>
>> --Sheng
>>>
>>> Wido
Re: [ACS41] Management server closes connection on port 8250
Posted by Sheng Yang <sh...@yasker.org>.
On Tue, Apr 2, 2013 at 11:49 AM, Sheng Yang <sh...@yasker.org> wrote:
> On Tue, Apr 2, 2013 at 11:32 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>> Hi,
>>
>> Since I've upgraded my management server to 4.1 my agents refuse to connect,
>> their log says:
>>
>> 2013-04-02 20:26:11,207 INFO [utils.nio.NioClient] (Agent-Selector:null)
>> Connecting to 31.25.X.X:8250
>> 2013-04-02 20:26:11,209 ERROR [utils.nio.NioConnection]
>> (Agent-Selector:null) Unable to initialize the threads.
>> java.io.IOException: SSL: Fail to init SSL! java.io.IOException: Connection
>> closed with -1 on reading size.
>> at com.cloud.utils.nio.NioClient.init(NioClient.java:83)
>> at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
>> at java.lang.Thread.run(Thread.java:679)
>>
>> So I tried a telnet connection:
>>
>> Connection closed by foreign host.
>> root@stack01:~# telnet 31.25.X.X 8250
>> Trying 31.25.X.X...
>> Connected to 31.X.X.X.
>> Escape character is '^]'.
>> Connection closed by foreign host.
>> root@stack01:~#
>>
>> So I didn't close the connection, but it was the management server.
>>
>> I cranked up the logging to DEBUG, but nothing shows in any of the logs, so
>> I have no clue why this isn't working.
>>
>> On the mgmt server I see Java in LISTEN state on port 8250
>>
>> There is no firewall on the management server (it's my lab!).
>>
>> Any clues to what this could be before I start filing a in Jira? Since I'm
>> not sure if this is a bug.
>
> Have you upgrade your agent?
Also you could try to enable TRACE for com.cloud.utils.nio to see more
log of NIO.
--Sheng
>
> --Sheng
>>
>> Wido
Re: [ACS41] Management server closes connection on port 8250
Posted by Sheng Yang <sh...@yasker.org>.
On Tue, Apr 2, 2013 at 11:32 AM, Wido den Hollander <wi...@widodh.nl> wrote:
> Hi,
>
> Since I've upgraded my management server to 4.1 my agents refuse to connect,
> their log says:
>
> 2013-04-02 20:26:11,207 INFO [utils.nio.NioClient] (Agent-Selector:null)
> Connecting to 31.25.X.X:8250
> 2013-04-02 20:26:11,209 ERROR [utils.nio.NioConnection]
> (Agent-Selector:null) Unable to initialize the threads.
> java.io.IOException: SSL: Fail to init SSL! java.io.IOException: Connection
> closed with -1 on reading size.
> at com.cloud.utils.nio.NioClient.init(NioClient.java:83)
> at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
> at java.lang.Thread.run(Thread.java:679)
>
> So I tried a telnet connection:
>
> Connection closed by foreign host.
> root@stack01:~# telnet 31.25.X.X 8250
> Trying 31.25.X.X...
> Connected to 31.X.X.X.
> Escape character is '^]'.
> Connection closed by foreign host.
> root@stack01:~#
>
> So I didn't close the connection, but it was the management server.
>
> I cranked up the logging to DEBUG, but nothing shows in any of the logs, so
> I have no clue why this isn't working.
>
> On the mgmt server I see Java in LISTEN state on port 8250
>
> There is no firewall on the management server (it's my lab!).
>
> Any clues to what this could be before I start filing a in Jira? Since I'm
> not sure if this is a bug.
Have you upgrade your agent?
--Sheng
>
> Wido