You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by Rakesh R <ra...@huawei.com> on 2012/06/01 10:42:41 UTC
BKClient is failing when zkclient connection delays
Hi,
I'm using the latest BookKeeper.4.1.0 branch and facing ZK ConnectionLossException:
Scenario:
-----------
I have started the ZK cluster and when tries to create a BookKeeper client from my application, it is throwing following ZooKeeper ConnectionLossException and is exitting.
12/06/01 11:44:31 WARN client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
12/06/01 11:44:31 INFO client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
12/06/01 11:44:31 INFO zookeeper.ClientCnxn: Client session timed out, have not heard from server in 4540ms for sessionid 0x0, closing socket connection and attempting reconnect
12/06/01 11:44:32 INFO zookeeper.ClientCnxn: Opening socket connection to server /10.18.40.91:2182
Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.bookkeeper.client.BookieWatcher.readBookiesBlocking(BookieWatcher.java:151)
at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:139)
at BKClient1.main(BKClient1.java:40)
12/06/01 11:44:36 WARN client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
When analyzed more, the root cause is:
BookKeeper.java:
--------------------
In the constructor of BookKeeper, immediately after creating the ZK client , it is going to bookieWatcher.readBookiesBlocking() for available bookies from ZK server before reaching SyncConnected event.
I think, we would properly use the existing countdown latch and wait till ZooKeeper client connection establishment before continue reading Bookies.
What others opinion?
Thanks,
Rakesh
Re: BKClient is failing when zkclient connection delays
Posted by Ivan Kelly <iv...@apache.org>.
I created BOOKKEEPER-281 for this.
I have a fix already, but want to make sure with a test case.
-Ivan
On Fri, Jun 01, 2012 at 11:19:37AM +0200, Flavio Junqueira wrote:
> Great catch, Rakesh. I suggest we cancel this release candidate and fix it for the next.
>
> -Flavio
>
> On Jun 1, 2012, at 10:42 AM, Rakesh R wrote:
>
> > Hi,
> >
> >
> >
> > I'm using the latest BookKeeper.4.1.0 branch and facing ZK ConnectionLossException:
> >
> >
> >
> > Scenario:
> >
> > -----------
> >
> > I have started the ZK cluster and when tries to create a BookKeeper client from my application, it is throwing following ZooKeeper ConnectionLossException and is exitting.
> >
> >
> >
> > 12/06/01 11:44:31 WARN client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
> > 12/06/01 11:44:31 INFO client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
> > 12/06/01 11:44:31 INFO zookeeper.ClientCnxn: Client session timed out, have not heard from server in 4540ms for sessionid 0x0, closing socket connection and attempting reconnect
> > 12/06/01 11:44:32 INFO zookeeper.ClientCnxn: Opening socket connection to server /10.18.40.91:2182
> > Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
> > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> > at org.apache.bookkeeper.client.BookieWatcher.readBookiesBlocking(BookieWatcher.java:151)
> > at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:139)
> > at BKClient1.main(BKClient1.java:40)
> > 12/06/01 11:44:36 WARN client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
> >
> >
> >
> > When analyzed more, the root cause is:
> >
> > BookKeeper.java:
> > --------------------
> > In the constructor of BookKeeper, immediately after creating the ZK client , it is going to bookieWatcher.readBookiesBlocking() for available bookies from ZK server before reaching SyncConnected event.
> >
> > I think, we would properly use the existing countdown latch and wait till ZooKeeper client connection establishment before continue reading Bookies.
> >
> > What others opinion?
> >
> > Thanks,
> > Rakesh
>
> flavio
> junqueira
> senior research scientist
>
> fpj@yahoo-inc.com
> direct +34 93-183-8828
>
> avinguda diagonal 177, 8th floor, barcelona, 08018, es
> phone (408) 349 3300 fax (408) 349 3301
>
RE: BKClient is failing when zkclient connection delays
Posted by Rakesh R <ra...@huawei.com>.
Thanks, its great !
________________________________________
From: Sijie Guo [guosijie@gmail.com]
Sent: Friday, June 01, 2012 3:19 PM
To: bookkeeper-dev@zookeeper.apache.org
Subject: Re: BKClient is failing when zkclient connection delays
yes. would include BK-279 for it.
On Fri, Jun 1, 2012 at 5:35 PM, Uma Maheswara Rao G <ma...@huawei.com>wrote:
> Thanks Flavio,
> This will effect Namenode startup.
>
> +1 for including it now.
>
> If that is the case, we can include BK-279 also. Because, When we run BKJM
> tests, we are facing that issue frequently.
>
> Since this is a problem of delay for connection establistment, some envs
> may face this issue consistently, that will be bad.
>
> Regards,
> Uma
> ________________________________________
> From: Flavio Junqueira [fpj@yahoo-inc.com]
> Sent: Friday, June 01, 2012 2:49 PM
> To: bookkeeper-dev@zookeeper.apache.org
> Subject: Re: BKClient is failing when zkclient connection delays
>
> Great catch, Rakesh. I suggest we cancel this release candidate and fix it
> for the next.
>
> -Flavio
>
> On Jun 1, 2012, at 10:42 AM, Rakesh R wrote:
>
> > Hi,
> >
> >
> >
> > I'm using the latest BookKeeper.4.1.0 branch and facing ZK
> ConnectionLossException:
> >
> >
> >
> > Scenario:
> >
> > -----------
> >
> > I have started the ZK cluster and when tries to create a BookKeeper
> client from my application, it is throwing following ZooKeeper
> ConnectionLossException and is exitting.
> >
> >
> >
> > 12/06/01 11:44:31 WARN client.ZooKeeperSaslClient: SecurityException:
> java.lang.SecurityException: Unable to locate a login configuration
> occurred when trying to find JAAS configuration.
> > 12/06/01 11:44:31 INFO client.ZooKeeperSaslClient: Client will not
> SASL-authenticate because the default JAAS configuration section 'Client'
> could not be found. If you are not using SASL, you may ignore this. On the
> other hand, if you expected SASL to work, please fix your JAAS
> configuration.
> > 12/06/01 11:44:31 INFO zookeeper.ClientCnxn: Client session timed out,
> have not heard from server in 4540ms for sessionid 0x0, closing socket
> connection and attempting reconnect
> > 12/06/01 11:44:32 INFO zookeeper.ClientCnxn: Opening socket connection
> to server /10.18.40.91:2182
> > Exception in thread "main"
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss
> > at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> > at
> org.apache.bookkeeper.client.BookieWatcher.readBookiesBlocking(BookieWatcher.java:151)
> > at
> org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:139)
> > at BKClient1.main(BKClient1.java:40)
> > 12/06/01 11:44:36 WARN client.ZooKeeperSaslClient: SecurityException:
> java.lang.SecurityException: Unable to locate a login configuration
> occurred when trying to find JAAS configuration.
> >
> >
> >
> > When analyzed more, the root cause is:
> >
> > BookKeeper.java:
> > --------------------
> > In the constructor of BookKeeper, immediately after creating the ZK
> client , it is going to bookieWatcher.readBookiesBlocking() for available
> bookies from ZK server before reaching SyncConnected event.
> >
> > I think, we would properly use the existing countdown latch and wait
> till ZooKeeper client connection establishment before continue reading
> Bookies.
> >
> > What others opinion?
> >
> > Thanks,
> > Rakesh
>
> flavio
> junqueira
> senior research scientist
>
> fpj@yahoo-inc.com
> direct +34 93-183-8828
>
> avinguda diagonal 177, 8th floor, barcelona, 08018, es
> phone (408) 349 3300 fax (408) 349 3301
>
Re: BKClient is failing when zkclient connection delays
Posted by Sijie Guo <gu...@gmail.com>.
yes. would include BK-279 for it.
On Fri, Jun 1, 2012 at 5:35 PM, Uma Maheswara Rao G <ma...@huawei.com>wrote:
> Thanks Flavio,
> This will effect Namenode startup.
>
> +1 for including it now.
>
> If that is the case, we can include BK-279 also. Because, When we run BKJM
> tests, we are facing that issue frequently.
>
> Since this is a problem of delay for connection establistment, some envs
> may face this issue consistently, that will be bad.
>
> Regards,
> Uma
> ________________________________________
> From: Flavio Junqueira [fpj@yahoo-inc.com]
> Sent: Friday, June 01, 2012 2:49 PM
> To: bookkeeper-dev@zookeeper.apache.org
> Subject: Re: BKClient is failing when zkclient connection delays
>
> Great catch, Rakesh. I suggest we cancel this release candidate and fix it
> for the next.
>
> -Flavio
>
> On Jun 1, 2012, at 10:42 AM, Rakesh R wrote:
>
> > Hi,
> >
> >
> >
> > I'm using the latest BookKeeper.4.1.0 branch and facing ZK
> ConnectionLossException:
> >
> >
> >
> > Scenario:
> >
> > -----------
> >
> > I have started the ZK cluster and when tries to create a BookKeeper
> client from my application, it is throwing following ZooKeeper
> ConnectionLossException and is exitting.
> >
> >
> >
> > 12/06/01 11:44:31 WARN client.ZooKeeperSaslClient: SecurityException:
> java.lang.SecurityException: Unable to locate a login configuration
> occurred when trying to find JAAS configuration.
> > 12/06/01 11:44:31 INFO client.ZooKeeperSaslClient: Client will not
> SASL-authenticate because the default JAAS configuration section 'Client'
> could not be found. If you are not using SASL, you may ignore this. On the
> other hand, if you expected SASL to work, please fix your JAAS
> configuration.
> > 12/06/01 11:44:31 INFO zookeeper.ClientCnxn: Client session timed out,
> have not heard from server in 4540ms for sessionid 0x0, closing socket
> connection and attempting reconnect
> > 12/06/01 11:44:32 INFO zookeeper.ClientCnxn: Opening socket connection
> to server /10.18.40.91:2182
> > Exception in thread "main"
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss
> > at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> > at
> org.apache.bookkeeper.client.BookieWatcher.readBookiesBlocking(BookieWatcher.java:151)
> > at
> org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:139)
> > at BKClient1.main(BKClient1.java:40)
> > 12/06/01 11:44:36 WARN client.ZooKeeperSaslClient: SecurityException:
> java.lang.SecurityException: Unable to locate a login configuration
> occurred when trying to find JAAS configuration.
> >
> >
> >
> > When analyzed more, the root cause is:
> >
> > BookKeeper.java:
> > --------------------
> > In the constructor of BookKeeper, immediately after creating the ZK
> client , it is going to bookieWatcher.readBookiesBlocking() for available
> bookies from ZK server before reaching SyncConnected event.
> >
> > I think, we would properly use the existing countdown latch and wait
> till ZooKeeper client connection establishment before continue reading
> Bookies.
> >
> > What others opinion?
> >
> > Thanks,
> > Rakesh
>
> flavio
> junqueira
> senior research scientist
>
> fpj@yahoo-inc.com
> direct +34 93-183-8828
>
> avinguda diagonal 177, 8th floor, barcelona, 08018, es
> phone (408) 349 3300 fax (408) 349 3301
>
RE: BKClient is failing when zkclient connection delays
Posted by Uma Maheswara Rao G <ma...@huawei.com>.
Thanks Flavio,
This will effect Namenode startup.
+1 for including it now.
If that is the case, we can include BK-279 also. Because, When we run BKJM tests, we are facing that issue frequently.
Since this is a problem of delay for connection establistment, some envs may face this issue consistently, that will be bad.
Regards,
Uma
________________________________________
From: Flavio Junqueira [fpj@yahoo-inc.com]
Sent: Friday, June 01, 2012 2:49 PM
To: bookkeeper-dev@zookeeper.apache.org
Subject: Re: BKClient is failing when zkclient connection delays
Great catch, Rakesh. I suggest we cancel this release candidate and fix it for the next.
-Flavio
On Jun 1, 2012, at 10:42 AM, Rakesh R wrote:
> Hi,
>
>
>
> I'm using the latest BookKeeper.4.1.0 branch and facing ZK ConnectionLossException:
>
>
>
> Scenario:
>
> -----------
>
> I have started the ZK cluster and when tries to create a BookKeeper client from my application, it is throwing following ZooKeeper ConnectionLossException and is exitting.
>
>
>
> 12/06/01 11:44:31 WARN client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
> 12/06/01 11:44:31 INFO client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
> 12/06/01 11:44:31 INFO zookeeper.ClientCnxn: Client session timed out, have not heard from server in 4540ms for sessionid 0x0, closing socket connection and attempting reconnect
> 12/06/01 11:44:32 INFO zookeeper.ClientCnxn: Opening socket connection to server /10.18.40.91:2182
> Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.bookkeeper.client.BookieWatcher.readBookiesBlocking(BookieWatcher.java:151)
> at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:139)
> at BKClient1.main(BKClient1.java:40)
> 12/06/01 11:44:36 WARN client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
>
>
>
> When analyzed more, the root cause is:
>
> BookKeeper.java:
> --------------------
> In the constructor of BookKeeper, immediately after creating the ZK client , it is going to bookieWatcher.readBookiesBlocking() for available bookies from ZK server before reaching SyncConnected event.
>
> I think, we would properly use the existing countdown latch and wait till ZooKeeper client connection establishment before continue reading Bookies.
>
> What others opinion?
>
> Thanks,
> Rakesh
flavio
junqueira
senior research scientist
fpj@yahoo-inc.com
direct +34 93-183-8828
avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300 fax (408) 349 3301
Re: BKClient is failing when zkclient connection delays
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
Great catch, Rakesh. I suggest we cancel this release candidate and fix it for the next.
-Flavio
On Jun 1, 2012, at 10:42 AM, Rakesh R wrote:
> Hi,
>
>
>
> I'm using the latest BookKeeper.4.1.0 branch and facing ZK ConnectionLossException:
>
>
>
> Scenario:
>
> -----------
>
> I have started the ZK cluster and when tries to create a BookKeeper client from my application, it is throwing following ZooKeeper ConnectionLossException and is exitting.
>
>
>
> 12/06/01 11:44:31 WARN client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
> 12/06/01 11:44:31 INFO client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
> 12/06/01 11:44:31 INFO zookeeper.ClientCnxn: Client session timed out, have not heard from server in 4540ms for sessionid 0x0, closing socket connection and attempting reconnect
> 12/06/01 11:44:32 INFO zookeeper.ClientCnxn: Opening socket connection to server /10.18.40.91:2182
> Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.bookkeeper.client.BookieWatcher.readBookiesBlocking(BookieWatcher.java:151)
> at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:139)
> at BKClient1.main(BKClient1.java:40)
> 12/06/01 11:44:36 WARN client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
>
>
>
> When analyzed more, the root cause is:
>
> BookKeeper.java:
> --------------------
> In the constructor of BookKeeper, immediately after creating the ZK client , it is going to bookieWatcher.readBookiesBlocking() for available bookies from ZK server before reaching SyncConnected event.
>
> I think, we would properly use the existing countdown latch and wait till ZooKeeper client connection establishment before continue reading Bookies.
>
> What others opinion?
>
> Thanks,
> Rakesh
flavio
junqueira
senior research scientist
fpj@yahoo-inc.com
direct +34 93-183-8828
avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300 fax (408) 349 3301
Re: BKClient is failing when zkclient connection delays
Posted by Sijie Guo <gu...@gmail.com>.
Thanks Rakesh.
Would cancel the release candidate until fixing this issue.
On Fri, Jun 1, 2012 at 4:42 PM, Rakesh R <ra...@huawei.com> wrote:
> Hi,
>
>
>
> I'm using the latest BookKeeper.4.1.0 branch and facing ZK
> ConnectionLossException:
>
>
>
> Scenario:
>
> -----------
>
> I have started the ZK cluster and when tries to create a BookKeeper client
> from my application, it is throwing following ZooKeeper
> ConnectionLossException and is exitting.
>
>
>
> 12/06/01 11:44:31 WARN client.ZooKeeperSaslClient: SecurityException:
> java.lang.SecurityException: Unable to locate a login configuration
> occurred when trying to find JAAS configuration.
> 12/06/01 11:44:31 INFO client.ZooKeeperSaslClient: Client will not
> SASL-authenticate because the default JAAS configuration section 'Client'
> could not be found. If you are not using SASL, you may ignore this. On the
> other hand, if you expected SASL to work, please fix your JAAS
> configuration.
> 12/06/01 11:44:31 INFO zookeeper.ClientCnxn: Client session timed out,
> have not heard from server in 4540ms for sessionid 0x0, closing socket
> connection and attempting reconnect
> 12/06/01 11:44:32 INFO zookeeper.ClientCnxn: Opening socket connection to
> server /10.18.40.91:2182
> Exception in thread "main"
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.bookkeeper.client.BookieWatcher.readBookiesBlocking(BookieWatcher.java:151)
> at
> org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:139)
> at BKClient1.main(BKClient1.java:40)
> 12/06/01 11:44:36 WARN client.ZooKeeperSaslClient: SecurityException:
> java.lang.SecurityException: Unable to locate a login configuration
> occurred when trying to find JAAS configuration.
>
>
>
> When analyzed more, the root cause is:
>
> BookKeeper.java:
> --------------------
> In the constructor of BookKeeper, immediately after creating the ZK client
> , it is going to bookieWatcher.readBookiesBlocking() for available bookies
> from ZK server before reaching SyncConnected event.
>
> I think, we would properly use the existing countdown latch and wait till
> ZooKeeper client connection establishment before continue reading Bookies.
>
> What others opinion?
>
> Thanks,
> Rakesh
>