You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Petersen, Robert" <ro...@mail.rakuten.com> on 2014/02/28 22:11:15 UTC
network slows when solr is running - help
Hi guys,
Got an odd thing going on right now. Indexing into my master server (solr 3.6.1) has slowed and it is because when solr runs ping shows latency. When I stop solr though, ping returns to normal. This has been happening occasionally, rebooting didn't help. This is the first time I noticed that stopping solr returns ping speeds to normal. I was thinking it was something with our network. Solr is not consuming all resources on the box or anything like that, and normally everything works fine. Has anyone seen this type of thing before? Let me know if more info of any kind is needed.
Solr process is at 8% memory utilization and 35% cpu utilization in 'top' command.
Note: solr is the only thing running on the box.
C:\Users\robertpe>ping 10.12.132.101 <-- Indexing
Pinging 10.12.132.101 with 32 bytes of data:
Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
Ping statistics for 10.12.132.101:
Packets: Sent = 4, Received = 4, Lost = 0 (0% lo
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
C:\Users\robertpe>ping 10.12.132.101 <-- Solr stopped
Pinging 10.12.132.101 with 32 bytes of data:
Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
Ping statistics for 10.12.132.101:
Packets: Sent = 4, Received = 4, Lost = 0 (0% lo
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
C:\Users\robertpe>ping 10.12.132.101 <-- Solr started but no indexing activity
Pinging 10.12.132.101 with 32 bytes of data:
Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
Ping statistics for 10.12.132.101:
Packets: Sent = 4, Received = 4, Lost = 0 (0% lo
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
C:\Users\robertpe>ping 10.12.132.101 <-- Solr started and indexing started
Pinging 10.12.132.101 with 32 bytes of data:
Reply from 10.12.132.101: bytes=32 time=53ms TTL=64
Reply from 10.12.132.101: bytes=32 time=51ms TTL=64
Reply from 10.12.132.101: bytes=32 time=48ms TTL=64
Reply from 10.12.132.101: bytes=32 time=51ms TTL=64
Ping statistics for 10.12.132.101:
Packets: Sent = 4, Received = 4, Lost = 0 (0% lo
Approximate round trip times in milli-seconds:
Minimum = 48ms, Maximum = 53ms, Average = 50ms
Robert (Robi) Petersen
Senior Software Engineer
Search Department
RE: network slows when solr is running - help
Posted by "Petersen, Robert" <ro...@mail.rakuten.com>.
<autoCommit>
<maxDocs>250000</maxDocs>
<maxTime>900000</maxTime>
</autoCommit>
-----Original Message-----
From: Lan [mailto:dung.lan@gmail.com]
Sent: Monday, March 03, 2014 1:24 PM
To: solr-user@lucene.apache.org
Subject: Re: network slows when solr is running - help
How frequently are you committing? Frequent commits can slow everything down.
--
View this message in context: http://lucene.472066.n3.nabble.com/network-slows-when-solr-is-running-help-tp4120523p4120992.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: network slows when solr is running - help
Posted by Lan <du...@gmail.com>.
How frequently are you committing? Frequent commits can slow everything down.
--
View this message in context: http://lucene.472066.n3.nabble.com/network-slows-when-solr-is-running-help-tp4120523p4120992.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: network slows when solr is running - help
Posted by "Petersen, Robert" <ro...@mail.rakuten.com>.
Yes my indexer runs as a service on a different box, it has 24 threads pushing docs to solr atomically. No the solr master is not virtual, it has 64 GB main memory and dual quad xeon cpus. The cpu utilization is not maxed out from what I can see in 'top'. Right now it says 38%. The other thing is that this only happens intermittently. I'm going to have IT update firmware on the NIC and then we'll open a ticket with HP for lack of anything else.
Here is some other information:
OS Name: Linux OS Version: 2.6.18-128.el5 Total RAM: 62.92 GB Free RAM: 44.20 GB Committed JVM memory: 36.03 GB Total swap: 20.00 GB Free swap: 20.00 GB
NUMBER OF REQUESTS EACH INTERVAL request count: 232678 error count: 5
PROCESSING TIME (MS) IN EACH INTERVAL processing time: 15355740 max time: 79408
TRAFFIC VOLUME (BYTES) IN EACH INTERVAL sent: 702 GB received: 956 MB
-----Original Message-----
From: Josh [mailto:jwdavey@gmail.com]
Sent: Friday, February 28, 2014 1:27 PM
To: solr-user@lucene.apache.org
Subject: Re: network slows when solr is running - help
Is it indexing data from over the network? (high data throughput would increase latency) Is it a virtual machine? (Other machines causing slow
downs) Another possible option is the network card is offloading processing onto the CPU which is introducing latency when the CPU is under load.
On Fri, Feb 28, 2014 at 4:11 PM, Petersen, Robert < robert.petersen@mail.rakuten.com> wrote:
> Hi guys,
>
> Got an odd thing going on right now. Indexing into my master server
> (solr
> 3.6.1) has slowed and it is because when solr runs ping shows latency.
> When I stop solr though, ping returns to normal. This has been
> happening occasionally, rebooting didn't help. This is the first time
> I noticed that stopping solr returns ping speeds to normal. I was thinking it was
> something with our network. Solr is not consuming all resources on the
> box or anything like that, and normally everything works fine. Has
> anyone seen this type of thing before? Let me know if more info of
> any kind is needed.
>
> Solr process is at 8% memory utilization and 35% cpu utilization in 'top'
> command.
>
> Note: solr is the only thing running on the box.
>
> C:\Users\robertpe>ping 10.12.132.101 <-- Indexing
>
> Pinging 10.12.132.101 with 32 bytes of data:
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64 Reply from
> 10.12.132.101: bytes=32 time<1ms TTL=64 Reply from 10.12.132.101:
> bytes=32 time<1ms TTL=64 Reply from 10.12.132.101: bytes=32 time<1ms
> TTL=64
>
> Ping statistics for 10.12.132.101:
> Packets: Sent = 4, Received = 4, Lost = 0 (0% lo Approximate round
> trip times in milli-seconds:
> Minimum = 0ms, Maximum = 0ms, Average = 0ms
>
> C:\Users\robertpe>ping 10.12.132.101 <-- Solr stopped
>
> Pinging 10.12.132.101 with 32 bytes of data:
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64 Reply from
> 10.12.132.101: bytes=32 time<1ms TTL=64 Reply from 10.12.132.101:
> bytes=32 time<1ms TTL=64 Reply from 10.12.132.101: bytes=32 time<1ms
> TTL=64
>
> Ping statistics for 10.12.132.101:
> Packets: Sent = 4, Received = 4, Lost = 0 (0% lo Approximate round
> trip times in milli-seconds:
> Minimum = 0ms, Maximum = 0ms, Average = 0ms
>
> C:\Users\robertpe>ping 10.12.132.101 <-- Solr started but no indexing
> activity
>
> Pinging 10.12.132.101 with 32 bytes of data:
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64 Reply from
> 10.12.132.101: bytes=32 time<1ms TTL=64 Reply from 10.12.132.101:
> bytes=32 time<1ms TTL=64 Reply from 10.12.132.101: bytes=32 time<1ms
> TTL=64
>
> Ping statistics for 10.12.132.101:
> Packets: Sent = 4, Received = 4, Lost = 0 (0% lo Approximate round
> trip times in milli-seconds:
> Minimum = 0ms, Maximum = 0ms, Average = 0ms
>
> C:\Users\robertpe>ping 10.12.132.101 <-- Solr started and indexing
> started
>
> Pinging 10.12.132.101 with 32 bytes of data:
> Reply from 10.12.132.101: bytes=32 time=53ms TTL=64 Reply from
> 10.12.132.101: bytes=32 time=51ms TTL=64 Reply from 10.12.132.101:
> bytes=32 time=48ms TTL=64 Reply from 10.12.132.101: bytes=32 time=51ms
> TTL=64
>
> Ping statistics for 10.12.132.101:
> Packets: Sent = 4, Received = 4, Lost = 0 (0% lo Approximate round
> trip times in milli-seconds:
> Minimum = 48ms, Maximum = 53ms, Average = 50ms
>
> Robert (Robi) Petersen
> Senior Software Engineer
> Search Department
>
>
>
>
Re: network slows when solr is running - help
Posted by Josh <jw...@gmail.com>.
Is it indexing data from over the network? (high data throughput would
increase latency) Is it a virtual machine? (Other machines causing slow
downs) Another possible option is the network card is offloading processing
onto the CPU which is introducing latency when the CPU is under load.
On Fri, Feb 28, 2014 at 4:11 PM, Petersen, Robert <
robert.petersen@mail.rakuten.com> wrote:
> Hi guys,
>
> Got an odd thing going on right now. Indexing into my master server (solr
> 3.6.1) has slowed and it is because when solr runs ping shows latency.
> When I stop solr though, ping returns to normal. This has been happening
> occasionally, rebooting didn't help. This is the first time I noticed that
> stopping solr returns ping speeds to normal. I was thinking it was
> something with our network. Solr is not consuming all resources on the
> box or anything like that, and normally everything works fine. Has anyone
> seen this type of thing before? Let me know if more info of any kind is
> needed.
>
> Solr process is at 8% memory utilization and 35% cpu utilization in 'top'
> command.
>
> Note: solr is the only thing running on the box.
>
> C:\Users\robertpe>ping 10.12.132.101 <-- Indexing
>
> Pinging 10.12.132.101 with 32 bytes of data:
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
>
> Ping statistics for 10.12.132.101:
> Packets: Sent = 4, Received = 4, Lost = 0 (0% lo
> Approximate round trip times in milli-seconds:
> Minimum = 0ms, Maximum = 0ms, Average = 0ms
>
> C:\Users\robertpe>ping 10.12.132.101 <-- Solr stopped
>
> Pinging 10.12.132.101 with 32 bytes of data:
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
>
> Ping statistics for 10.12.132.101:
> Packets: Sent = 4, Received = 4, Lost = 0 (0% lo
> Approximate round trip times in milli-seconds:
> Minimum = 0ms, Maximum = 0ms, Average = 0ms
>
> C:\Users\robertpe>ping 10.12.132.101 <-- Solr started but no indexing
> activity
>
> Pinging 10.12.132.101 with 32 bytes of data:
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
> Reply from 10.12.132.101: bytes=32 time<1ms TTL=64
>
> Ping statistics for 10.12.132.101:
> Packets: Sent = 4, Received = 4, Lost = 0 (0% lo
> Approximate round trip times in milli-seconds:
> Minimum = 0ms, Maximum = 0ms, Average = 0ms
>
> C:\Users\robertpe>ping 10.12.132.101 <-- Solr started and indexing started
>
> Pinging 10.12.132.101 with 32 bytes of data:
> Reply from 10.12.132.101: bytes=32 time=53ms TTL=64
> Reply from 10.12.132.101: bytes=32 time=51ms TTL=64
> Reply from 10.12.132.101: bytes=32 time=48ms TTL=64
> Reply from 10.12.132.101: bytes=32 time=51ms TTL=64
>
> Ping statistics for 10.12.132.101:
> Packets: Sent = 4, Received = 4, Lost = 0 (0% lo
> Approximate round trip times in milli-seconds:
> Minimum = 48ms, Maximum = 53ms, Average = 50ms
>
> Robert (Robi) Petersen
> Senior Software Engineer
> Search Department
>
>
>
>