You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Santanu8939967892 <mi...@gmail.com> on 2013/08/20 08:10:23 UTC
indexing fails after 40/ 50 million data being indexed - single zookeeper
Hi,
We have a requirement of huge 250 million data to be indexed and we are
using DIH to index the data in batches of 10 million. We are using Solr 4.4.
We are using Solr cloud with One zookeeper and two collections with two
shards in each in two Tomcat servers.
But the indexing fails after 40/ 50 million data being indexed.
Please let me know how to mitigate this issue.
I am posting the zookeeper log below.
2013-08-20 00:36:32,188 [myid:] - INFO [SyncThread:0:ZooKeeperServer@595]
- Est
ablished session 0x1409a3945380000 with negotiated timeout 30000 for client
/0:0
:0:0:0:0:0:1:53440
2013-08-20 00:36:33,873 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@476] - Processed session termination for sessionid:
0x1409a39453800
00
2013-08-20 00:36:33,873 [myid:] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:21
81:NIOServerCnxn@1001] - Closed socket connection for client
/0:0:0:0:0:0:0:1:53
440 which had sessionid 0x1409a3945380000
2013-08-20 00:36:43,826 [myid:] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:21
81:NIOServerCnxnFactory@197] - Accepted socket connection from /
127.0.0.1:53449
2013-08-20 00:36:43,826 [myid:] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:21
81:ZooKeeperServer@839] - Client attempting to establish new session at
/127.0.0
.1:53449
2013-08-20 00:36:43,872 [myid:] - INFO [SyncThread:0:ZooKeeperServer@595]
- Est
ablished session 0x1409a3945380001 with negotiated timeout 30000 for client
/127
.0.0.1:53449
2013-08-20 00:36:44,013 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@476] - Processed session termination for sessionid:
0x1409a39453800
01
2013-08-20 00:36:44,028 [myid:] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:21
81:NIOServerCnxn@1001] - Closed socket connection for client /
127.0.0.1:53449 wh
ich had sessionid 0x1409a3945380001
2013-08-20 00:36:47,133 [myid:] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:21
81:NIOServerCnxnFactory@197] - Accepted socket connection from /
127.0.0.1:53455
2013-08-20 00:36:47,148 [myid:] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:21
81:ZooKeeperServer@839] - Client attempting to establish new session at
/127.0.0
.1:53455
2013-08-20 00:36:47,164 [myid:] - INFO [SyncThread:0:ZooKeeperServer@595]
- Est
ablished session 0x1409a3945380002 with negotiated timeout 30000 for client
/127
.0.0.1:53455
2013-08-20 00:36:49,754 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@476] - Processed session termination for sessionid:
0x1409a39453800
02
2013-08-20 00:36:49,769 [myid:] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:21
81:NIOServerCnxn@1001] - Closed socket connection for client /
127.0.0.1:53455 wh
ich had sessionid 0x1409a3945380002
2013-08-20 00:36:54,824 [myid:] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:21
81:NIOServerCnxnFactory@197] - Accepted socket connection from /
127.0.0.1:53464
2013-08-20 00:36:54,839 [myid:] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:21
81:ZooKeeperServer@839] - Client attempting to establish new session at
/127.0.0
.1:53464
2013-08-20 00:36:54,870 [myid:] - INFO [SyncThread:0:ZooKeeperServer@595]
- Est
ablished session 0x1409a3945380003 with negotiated timeout 30000 for client
/127
.0.0.1:53464
2013-08-20 00:36:55,011 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@476] - Processed session termination for sessionid:
0x1409a39453800
03
2013-08-20 00:36:55,058 [myid:] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:21
81:NIOServerCnxn@1001] - Closed socket connection for client /
127.0.0.1:53464 wh
ich had sessionid 0x1409a3945380003
2013-08-20 00:37:00,003 [myid:] - INFO [SessionTracker:ZooKeeperServer@325]
- E
xpiring session 0x1409617c5750005, timeout of 40000ms exceeded
2013-08-20 00:37:00,003 [myid:] - INFO [SessionTracker:ZooKeeperServer@325]
- E
xpiring session 0x1409617c5750004, timeout of 40000ms exceeded
2013-08-20 00:37:00,003 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@476] - Processed session termination for sessionid:
0x1409617c57500
05
2013-08-20 00:37:00,003 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@476] - Processed session termination for sessionid:
0x1409617c57500
04
2013-08-20 00:37:17,584 [myid:] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:21
81:NIOServerCnxnFactory@197] - Accepted socket connection from /
127.0.0.1:53480
2013-08-20 00:37:17,584 [myid:] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:21
81:ZooKeeperServer@839] - Client attempting to establish new session at
/127.0.0
.1:53480
2013-08-20 00:37:17,600 [myid:] - INFO [SyncThread:0:ZooKeeperServer@595]
- Est
ablished session 0x1409a3945380004 with negotiated timeout 40000 for client
/127
.0.0.1:53480
2013-08-20 00:37:17,631 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@627] - Got user-level KeeperException when processing
sessionid:0x1
409a3945380004 type:create cxid:0x1 zxid:0x91e txntype:-1 reqpath:n/a Error
Path
:/overseer Error:KeeperErrorCode = NodeExists for /overseer
2013-08-20 00:37:17,662 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@627] - Got user-level KeeperException when processing
sessionid:0x1
409a3945380004 type:create cxid:0x3 zxid:0x91f txntype:-1 reqpath:n/a Error
Path
:/overseer Error:KeeperErrorCode = NodeExists for /overseer
2013-08-20 00:37:48,176 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@627] - Got user-level KeeperException when processing
sessionid:0x1
409a3945380004 type:delete cxid:0x12 zxid:0x924 txntype:-1 reqpath:n/a
Error Pat
h:/live_nodes/localhost:8280_solr Error:KeeperErrorCode = NoNode for
/live_nodes
/localhost:8280_solr
2013-08-20 00:37:48,238 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@627] - Got user-level KeeperException when processing
sessionid:0x1
409a3945380004 type:delete cxid:0x1c zxid:0x927 txntype:-1 reqpath:n/a
Error Pat
h:/overseer_elect/leader Error:KeeperErrorCode = NoNode for
/overseer_elect/lead
er
2013-08-20 00:37:48,254 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@627] - Got user-level KeeperException when processing
sessionid:0x1
409a3945380004 type:create cxid:0x21 zxid:0x929 txntype:-1 reqpath:n/a
Error Pat
h:/overseer Error:KeeperErrorCode = NodeExists for /overseer
2013-08-20 00:37:48,285 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@627] - Got user-level KeeperException when processing
sessionid:0x1
409a3945380004 type:create cxid:0x22 zxid:0x92a txntype:-1 reqpath:n/a
Error Pat
h:/overseer Error:KeeperErrorCode = NodeExists for /overseer
2013-08-20 00:37:48,300 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@627] - Got user-level KeeperException when processing
sessionid:0x1
409a3945380004 type:create cxid:0x24 zxid:0x92b txntype:-1 reqpath:n/a
Error Pat
h:/overseer Error:KeeperErrorCode = NodeExists for /overseer
2013-08-20 00:37:48,316 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@627] - Got user-level KeeperException when processing
sessionid:0x1
409a3945380004 type:create cxid:0x26 zxid:0x92c txntype:-1 reqpath:n/a
Error Pat
h:/overseer Error:KeeperErrorCode = NodeExists for /overseer
2013-08-20 00:37:51,405 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@627] - Got user-level KeeperException when processing
sessionid:0x1
409a3945380004 type:delete cxid:0x223 zxid:0x949 txntype:-1 reqpath:n/a
Error Pa
th:/collections/firstcollection/leaders/shard1 Error:KeeperErrorCode =
NoNode fo
r /collections/firstcollection/leaders/shard1
2013-08-20 00:37:51,405 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@627] - Got user-level KeeperException when processing
sessionid:0x1
409a3945380004 type:delete cxid:0x224 zxid:0x94a txntype:-1 reqpath:n/a
Error Pa
th:/collections/firstcollection/leaders/shard2 Error:KeeperErrorCode =
NoNode fo
r /collections/firstcollection/leaders/shard2
2013-08-20 00:37:51,405 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@627] - Got user-level KeeperException when processing
sessionid:0x1
409a3945380004 type:delete cxid:0x225 zxid:0x94b txntype:-1 reqpath:n/a
Error Pa
th:/collections/secondcollection/leaders/shard2 Error:KeeperErrorCode =
NoNode f
or /collections/secondcollection/leaders/shard2
2013-08-20 00:37:51,405 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@627] - Got user-level KeeperException when processing
sessionid:0x1
409a3945380004 type:create cxid:0x226 zxid:0x94c txntype:-1 reqpath:n/a
Error Pa
th:/overseer Error:KeeperErrorCode = NodeExists for /overseer
2013-08-20 00:37:51,405 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@627] - Got user-level KeeperException when processing
sessionid:0x1
409a3945380004 type:create cxid:0x227 zxid:0x94d txntype:-1 reqpath:n/a
Error Pa
th:/overseer Error:KeeperErrorCode = NodeExists for /overseer
2013-08-20 00:37:51,405 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequ
estProcessor@627] - Got user-level KeeperException when processing
sessionid:0x1
409a3945380004 type:create cxid:0x229 zxid:0x94e txntype:-1 reqpath:n/a
Error Pa
th:/overseer Error:KeeperErrorCode = NodeExists for /overseer
With Regards,
Santanu