You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Reinhard Budenstecher <ra...@maills.de> on 2016/10/09 18:33:19 UTC
Config for massive inserts into Solr master
Hello,
I'm not a pro in Solr nor in Java, so please be patient. We have an ecommerce application with 150 millions docs and a size of 140GB in Solr.
We are using the following setup:
Solr "MASTER":
- DELL R530, 1x XEON E5-1650
- 64GB ECC RAM
- 4x 480GB SSD as RAID10 on hardware RAID (but no BBU so no writeback)
Solr "SLAVE":
- DELL R730, 2x XEON E5-2600
- 128GB ECC RAM
- 4x 480GB SSD as RAID10 on hardware RAID
Both systems are running with Debian Jessie, OpenJDK 8 and Solr 6.2.1. Modification to config, that are done:
- heap memory 32GB on MASTER and 64GB on slave
- disabled "AddSchemaFieldsUpdateProcessorFactory"
- set "ClassicIndexSchemaFactory"
- enabled "useFilterForSortedQuery"
- and of course: enabled replication
- autoCommit is 15 seconds and openSearcher is false
- rest in on stock
We have an ETL process which updates product catalog. This produces massive inserts on MASTER, but there are no reads. Often there are thousands and hundreds of thousands of records per minute that where inserted. But sometimes I get a an OOM error, the only log entry I can find is:
2016-10-09T16:17:34.440+0200: 63872,249: [Full GC (Allocation Failure) 2016-10-09T16:17:34.440+0200: 63872,249: [CMS: 16387099K->16387087K(16777216K), 4,2227778 secs] 17782619K->17782606K(30758272K), [Metaspace: 36452K->36452K(38912K)], 4,2229287 secs] [Times: user=4,22 sys=0,01, real=4,22 secs]
As I'm a bit lost with this all: is there anybody who can help me with best config for massive inserts on MASTER and massive reads on SLAVE. Is there a common approach? What details should I provide furthermore? Or is the simplest solution to raise heap on MASTER from 32GB of the available 64GB to a higher value?
______________________________________________________
Gesendet mit Maills.de - mehr als nur Freemail www.maills.de
Re: Re: Config for massive inserts into Solr master
Posted by Reinhard Budenstecher <ra...@maills.de>.
>
> What I have been hoping to see is the exact text of an OutOfMemoryError
> in solr.log so I can tell whether it's happening because of heap space
> or some other problem, like stack space. The stacktrace on such an
> error might be helpfultoo.
>
Hi,
I did understand what you need, I'm newbie to Solr and Java but not to Linux. So please believe me, the only data that gets written on OOM is
# solr_oom_killer-8983-2016-10-10_00_02_46.log
Running OOM killer script for process 116987 for Solr on port 8983
Killed process 116987
# solr.log
<long warning about temporary segment, that I've posted ago>
# solr_gc_log_20161010_0810
2016-10-10T00:02:26.331+0200: 25715,434: [GC (CMS Initial Mark) [1 CMS-initial-mark: 12114331K(16777216K)] 12163041K(30758272K), 0,0036996 secs] [Times: user=0,01 sys=0,00, real=0,01 secs]
2016-10-10T00:02:26.335+0200: 25715,438: Total time for which application threads were stopped: 0,1433940 seconds, Stopping threads took: 0,1393095 seconds
2016-10-10T00:02:26.335+0200: 25715,438: [CMS-concurrent-mark-start]
2016-10-10T00:02:26.594+0200: 25715,697: Total time for which application threads were stopped: 0,1226207 seconds, Stopping threads took: 0,1223130 seconds
2016-10-10T00:02:26.901+0200: 25716,003: Total time for which application threads were stopped: 0,3050990 seconds, Stopping threads took: 0,3047010 seconds
2016-10-10T00:02:26.960+0200: 25716,063: Total time for which application threads were stopped: 0,0043570 seconds, Stopping threads took: 0,0040603 seconds
2016-10-10T00:02:26.960+0200: 25716,063: Total time for which application threads were stopped: 0,0002308 seconds, Stopping threads took: 0,0000567 seconds
2016-10-10T00:02:28.324+0200: 25717,426: Total time for which application threads were stopped: 0,0061071 seconds, Stopping threads took: 0,0059022 seconds
2016-10-10T00:02:29.325+0200: 25718,428: Total time for which application threads were stopped: 0,0003410 seconds, Stopping threads took: 0,0001455 seconds
2016-10-10T00:02:31.056+0200: 25720,158: [CMS-concurrent-mark: 4,280/4,721 secs] [Times: user=21,41 sys=0,44, real=4,72 secs]
2016-10-10T00:02:31.056+0200: 25720,158: [CMS-concurrent-preclean-start]
2016-10-10T00:02:31.332+0200: 25720,434: Total time for which application threads were stopped: 0,0029760 seconds, Stopping threads took: 0,0027661 seconds
2016-10-10T00:02:32.187+0200: 25721,290: [CMS-concurrent-preclean: 0,965/1,131 secs] [Times: user=5,92 sys=0,09, real=1,13 secs]
2016-10-10T00:02:32.187+0200: 25721,290: [CMS-concurrent-abortable-preclean-start]
2016-10-10T00:02:32.546+0200: 25721,648: Total time for which application threads were stopped: 0,0006876 seconds, Stopping threads took: 0,0001229 seconds
2016-10-10T00:02:32.964+0200: 25722,066: Total time for which application threads were stopped: 0,0020954 seconds, Stopping threads took: 0,0018945 seconds
2016-10-10T00:02:33.794+0200: 25722,896: [CMS-concurrent-abortable-preclean: 1,573/1,607 secs] [Times: user=9,04 sys=0,12, real=1,61 secs]
2016-10-10T00:02:33.926+0200: 25723,028: [GC (CMS Final Remark) [YG occupancy: 4396269 K (13981056 K)]{Heap before GC invocations=478 (full 6):
par new generation total 13981056K, used 4396269K [0x00007fa0d8000000, 0x00007fa4d8000000, 0x00007fa4d8000000)
eden space 11184896K, 39% used [0x00007fa0d8000000, 0x00007fa1e453b598, 0x00007fa382ac0000)
from space 2796160K, 0% used [0x00007fa42d560000, 0x00007fa42d560000, 0x00007fa4d8000000)
to space 2796160K, 0% used [0x00007fa382ac0000, 0x00007fa382ac0000, 0x00007fa42d560000)
concurrent mark-sweep generation total 16777216K, used 15699719K [0x00007fa4d8000000, 0x00007fa8d8000000, 0x00007fa8d8000000)
Metaspace used 36740K, capacity 37441K, committed 37684K, reserved 38912K
2016-10-10T00:02:33.926+0200: 25723,028: [GC (CMS Final Remark) 2016-10-10T00:02:33.926+0200: 25723,029: [ParNew: 4396269K->4396269K(13981056K), 0,0000212 secs] 20095988K->20095988K(30758272K), 0,00
06292 secs] [Times: user=0,00 sys=0,00, real=0,00 secs]
Heap after GC invocations=479 (full 6):
par new generation total 13981056K, used 4396269K [0x00007fa0d8000000, 0x00007fa4d8000000, 0x00007fa4d8000000)
eden space 11184896K, 39% used [0x00007fa0d8000000, 0x00007fa1e453b598, 0x00007fa382ac0000)
from space 2796160K, 0% used [0x00007fa42d560000, 0x00007fa42d560000, 0x00007fa4d8000000)
to space 2796160K, 0% used [0x00007fa382ac0000, 0x00007fa382ac0000, 0x00007fa42d560000)
concurrent mark-sweep generation total 16777216K, used 15699719K [0x00007fa4d8000000, 0x00007fa8d8000000, 0x00007fa8d8000000)
Metaspace used 36740K, capacity 37441K, committed 37684K, reserved 38912K
}
2016-10-10T00:02:33.926+0200: 25723,029: [Rescan (parallel) , 1,4594466 secs]2016-10-10T00:02:35.386+0200: 25724,488: [weak refs processing, 0,0065564 secs]2016-10-10T00:02:35.392+0200: 25724,495: [class unloading, 0,0089755 secs]2016-10-10T00:02:35.401+0200: 25724,504: [scrub symbol table, 0,0044581 secs]2016-10-10T00:02:35.406+0200: 25724,508: [scrub string table, 0,0005908 secs][1 CMS-remark: 15699719K(16777216K)] 20095988K(30758272K), 1,4808925 secs] [Times: user=2,92 sys=0,01, real=1,48 secs]
2016-10-10T00:02:35.407+0200: 25724,509: Total time for which application threads were stopped: 1,6124523 seconds, Stopping threads took: 0,1312338 seconds
2016-10-10T00:02:35.407+0200: 25724,509: [CMS-concurrent-sweep-start]
2016-10-10T00:02:35.824+0200: 25724,926: [CMS-concurrent-sweep: 0,417/0,417 secs] [Times: user=1,86 sys=0,02, real=0,42 secs]
2016-10-10T00:02:35.824+0200: 25724,926: [CMS-concurrent-reset-start]
2016-10-10T00:02:35.862+0200: 25724,964: [CMS-concurrent-reset: 0,038/0,038 secs] [Times: user=0,15 sys=0,00, real=0,03 secs]
{Heap before GC invocations=479 (full 6):
{Heap before GC invocations=479 (full 6):
par new generation total 13981056K, used 5767902K [0x00007fa0d8000000, 0x00007fa4d8000000, 0x00007fa4d8000000)
eden space 11184896K, 51% used [0x00007fa0d8000000, 0x00007fa2380b7b28, 0x00007fa382ac0000)
from space 2796160K, 0% used [0x00007fa42d560000, 0x00007fa42d560000, 0x00007fa4d8000000)
to space 2796160K, 0% used [0x00007fa382ac0000, 0x00007fa382ac0000, 0x00007fa42d560000)
concurrent mark-sweep generation total 16777216K, used 15698847K [0x00007fa4d8000000, 0x00007fa8d8000000, 0x00007fa8d8000000)
Metaspace used 36740K, capacity 37441K, committed 37684K, reserved 38912K
2016-10-10T00:02:35.922+0200: 25725,024: [GC (Allocation Failure) 2016-10-10T00:02:35.922+0200: 25725,024: [CMS: 15698847K->15716846K(16777216K), 5,0241273 secs] 21466750K->18470223K(30758272K), [Metaspace: 36740K->36740K(38912K)], 5,0244793 secs] [Times: user=5,03 sys=0,00, real=5,03 secs]
Heap after GC invocations=480 (full 7):
par new generation total 13981056K, used 2753376K [0x00007fa0d8000000, 0x00007fa4d8000000, 0x00007fa4d8000000)
eden space 11184896K, 24% used [0x00007fa0d8000000, 0x00007fa1800d8238, 0x00007fa382ac0000)
from space 2796160K, 0% used [0x00007fa42d560000, 0x00007fa42d560000, 0x00007fa4d8000000)
to space 2796160K, 0% used [0x00007fa382ac0000, 0x00007fa382ac0000, 0x00007fa42d560000)
concurrent mark-sweep generation total 16777216K, used 15716846K [0x00007fa4d8000000, 0x00007fa8d8000000, 0x00007fa8d8000000)
Metaspace used 36740K, capacity 37441K, committed 37684K, reserved 38912K
}
{Heap before GC invocations=480 (full 7):
par new generation total 13981056K, used 2753376K [0x00007fa0d8000000, 0x00007fa4d8000000, 0x00007fa4d8000000)
eden space 11184896K, 24% used [0x00007fa0d8000000, 0x00007fa1800d8238, 0x00007fa382ac0000)
from space 2796160K, 0% used [0x00007fa42d560000, 0x00007fa42d560000, 0x00007fa4d8000000)
to space 2796160K, 0% used [0x00007fa382ac0000, 0x00007fa382ac0000, 0x00007fa42d560000)
concurrent mark-sweep generation total 16777216K, used 15716846K [0x00007fa4d8000000, 0x00007fa8d8000000, 0x00007fa8d8000000)
Metaspace used 36740K, capacity 37441K, committed 37684K, reserved 38912K
2016-10-10T00:02:40.946+0200: 25730,049: [Full GC (Allocation Failure) 2016-10-10T00:02:40.946+0200: 25730,049: [CMS: 15716846K->15716754K(16777216K), 4,6906650 secs] 18470223K->18470130K(30758272K), [Metaspace: 36740K->36740K(38912K)], 4,6907440 secs] [Times: user=4,69 sys=0,00, real=4,69 secs]
I'm sorry, that I cannot provide more informations.
______________________________________________________
Gesendet mit Maills.de - mehr als nur Freemail www.maills.de
Re: Config for massive inserts into Solr master
Posted by Shawn Heisey <ap...@elyograg.org>.
> Really, there is nothing in solr. log. I did not change any option
> related
> to this in config. Solr died again some hours ago and the last entry
> is: 2016-10-09 22:02:31.051 WARN (qtp225493257-1097) [ ]
> o.a.s.h.a.LukeRequestHandler Error getting file length for
> [segments_9102]
That's a warning message, not an error, and would not cause Solr to
crash. It's a common message on recent Solr versions, even when things
are working perfectly. We have some jira issues on it, I thought it had
been eliminated, but perhaps not.
Are there any logfiles with filenames that fit the following naming
scheme in your installation? The filename would include Solr's port
number and a timestamp.
solr_oom_killer-$SOLR_PORT-$NOW.log
If there are, then the times in those logfiles would correspond to the
Java process committing suicide because of an OutOfMemoryError. On
UNIX-like operating systems, the latest versions of Solr have a
functional OOM killer script. Killing Solr is the only reasonable
response to such an error, because an out of memory situation leaves a
java program in a completely unpredictable state where anything might
happen.
What I have been hoping to see is the exact text of an OutOfMemoryError
in solr.log so I can tell whether it's happening because of heap space
or some other problem, like stack space. The stacktrace on such an
error might be helpfultoo.
Thanks,
Shawn
Re: Config for massive inserts into Solr master
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/10/2016 2:49 AM, Alexandre Rafalovitch wrote:
> Just a sanity check. That directory mentioned, what kind of file
> system is that on? NFS, NAS, RAID?
The original post says it's hardware RAID10 with locally installed SSD
disks. It doesn't mention what filesystem is on it. If I were building
it, I would probably use ext4, or MAYBE install zfs on linux for the
Solr data. I like snapshots. :)
Thanks,
Shawn
Re: Sharding strategies
Posted by Reth RM <re...@gmail.com>.
If you will have numerous documents, splitting documents into shard is a
strategy. This split is independent of lingo of document.
For documents with different languages, its necessary to use language
specific analyzers to obtain good search results. For example, assume you
have english language documents, its _text_ field should ideally be
text_en; likewise, for Chinese/Japanese/Korean type documents, its fields'
fieldType should be text_cjk. If you mix documents of different language
type in same shard, then you will have to define multiple fieldTypes for
each language of document and also at query time manage, need to ensure to
query on respective fields.
There are different strategies that can be applied to have multilingual
search, slide 19 in this ppt explains them
http://www.slideshare.net/treygrainger/semantic-multilingual-strategies-in-lucenesolr
and
there's another article here based on the assumption that we know the
language of the incoming document and the language in which the query could
be
https://support.lucidworks.com/hc/en-us/articles/203718886-How-to-implement-Multilingual-Search-using-Solr
On Mon, Oct 10, 2016 at 8:08 AM, Customer <ma...@gmail.com> wrote:
> Hi,
>
>
> I'm started working on the project which will likely have lots of
> documents in every single language and because of that I'm a bit worried
> storing everything into one single shard. What would be the best way for
> data store, any advices how I should split my data ? I was thinking about
> going for alphabet (make a shard for every single alphabet letter, but
> knowing fact that there will be lots of languages - not only English, this
> is not an option).
>
> Thank you for your advicesin advance.
>
Sharding strategies
Posted by Customer <ma...@gmail.com>.
Hi,
I'm started working on the project which will likely have lots of
documents in every single language and because of that I'm a bit worried
storing everything into one single shard. What would be the best way for
data store, any advices how I should split my data ? I was thinking
about going for alphabet (make a shard for every single alphabet letter,
but knowing fact that there will be lots of languages - not only
English, this is not an option).
Thank you for your advicesin advance.
Re: Re: Re: Config for massive inserts into Solr master
Posted by Reinhard Budenstecher <ra...@maills.de>.
>
> Just a sanity check. That directory mentioned, what kind of file system is that on? NFS, NAS, RAID?
I'm using Ext4 with options "noatime,nodiratime,barrier=0" on a hardware RAID10 with 4 SSD disks
______________________________________________________
Gesendet mit Maills.de - mehr als nur Freemail www.maills.de
Re: Re: Config for massive inserts into Solr master
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Just a sanity check. That directory mentioned, what kind of file system is
that on? NFS, NAS, RAID?
Regards,
Alex
On 10 Oct 2016 1:09 AM, "Reinhard Budenstecher" <ra...@maills.de> wrote:
>
> That's considerably larger than you initially indicated. In just one
> index, you've got almost 300 million docs taking up well over 200GB.
> About half of them have been deleted, but they are still there. Those
> deleted docs *DO* affect operation and memory usage.
>
Yes, that's larger than I expected. Two days ago the index was at the size
I've written. This huge increase does happen because of running ETL.
>
> usage. The only effective way to get rid of them is to optimize the
> index ... but I will warn you that with an index of that size, the time
> required for an optimize can reach into multiple hours, and will
> temporarily require considerable additional disk space. The fact that
Three days ago we've upgraded from Solr 5.5.3 to 6.2.1. Before upgrading
I've optimized this index already and yes, it took some hours. So when two
days of ETL cause such an increase of index size, running a daily optimize
is not an option.
>
> You don't need to create it. Stacktraces are logged by Solr, in a file
> named solr.log, whenever most errors occur.
>
Really, there is nothing in solr. log. I did not change any option related
to this in config. Solr died again some hours ago and the last entry is:
2016-10-09 22:02:31.051 WARN (qtp225493257-1097) [ ]
o.a.s.h.a.LukeRequestHandler Error getting file length for [segments_9102]
java.nio.file.NoSuchFileException: /var/solr/data/myshop/data/
index/segments_9102
at sun.nio.fs.UnixException.translateToIOException(
UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(
UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(
UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(
UnixFileAttributeViews.java:55)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(
UnixFileSystemProvider.java:144)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(
LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.Files.size(Files.java:2332)
at org.apache.lucene.store.FSDirectory.fileLength(
FSDirectory.java:243)
at org.apache.lucene.store.NRTCachingDirectory.fileLength(
NRTCachingDirectory.java:128)
at org.apache.solr.handler.admin.LukeRequestHandler.getFileLength(
LukeRequestHandler.java:597)
at org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(
LukeRequestHandler.java:585)
at org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(
CoreAdminOperation.java:1007)
at org.apache.solr.handler.admin.CoreAdminOperation.lambda$
static$3(CoreAdminOperation.java:170)
at org.apache.solr.handler.admin.CoreAdminOperation.execute(
CoreAdminOperation.java:1056)
at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.
call(CoreAdminHandler.java:365)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(
CoreAdminHandler.java:156)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(
RequestHandlerBase.java:154)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(
HttpSolrCall.java:658)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:440)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
SolrDispatchFilter.java:257)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
SolrDispatchFilter.java:208)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(
ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(
ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(
SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.
doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.
doHandle(ContextHandler.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(
ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.
doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.
doScope(ContextHandler.java:1092)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(
ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.
handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(
HttpConnection.java:244)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
produceAndRun(ExecuteProduceConsume.java:246)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
So the only solution is to reduce index size by lowering number of docs or
reducing amount of indexed data per doc (which is not an option also)? We
can extend RAM to 192GB, but if that helps would only be a try. The numer
of "active" docs would remain at 150-160 million docs, but the number of
deleted docs caused by ETL process will not be forseeable.
______________________________________________________
Gesendet mit Maills.de - mehr als nur Freemail www.maills.de
Re: Re: Config for massive inserts into Solr master
Posted by Reinhard Budenstecher <ra...@maills.de>.
>
> That's considerably larger than you initially indicated. In just one
> index, you've got almost 300 million docs taking up well over 200GB.
> About half of them have been deleted, but they are still there. Those
> deleted docs *DO* affect operation and memory usage.
>
Yes, that's larger than I expected. Two days ago the index was at the size I've written. This huge increase does happen because of running ETL.
>
> usage. The only effective way to get rid of them is to optimize the
> index ... but I will warn you that with an index of that size, the time
> required for an optimize can reach into multiple hours, and will
> temporarily require considerable additional disk space. The fact that
Three days ago we've upgraded from Solr 5.5.3 to 6.2.1. Before upgrading I've optimized this index already and yes, it took some hours. So when two days of ETL cause such an increase of index size, running a daily optimize is not an option.
>
> You don't need to create it. Stacktraces are logged by Solr, in a file
> named solr.log, whenever most errors occur.
>
Really, there is nothing in solr. log. I did not change any option related to this in config. Solr died again some hours ago and the last entry is:
2016-10-09 22:02:31.051 WARN (qtp225493257-1097) [ ] o.a.s.h.a.LukeRequestHandler Error getting file length for [segments_9102]
java.nio.file.NoSuchFileException: /var/solr/data/myshop/data/index/segments_9102
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.Files.size(Files.java:2332)
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
at org.apache.lucene.store.NRTCachingDirectory.fileLength(NRTCachingDirectory.java:128)
at org.apache.solr.handler.admin.LukeRequestHandler.getFileLength(LukeRequestHandler.java:597)
at org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:585)
at org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:1007)
at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$3(CoreAdminOperation.java:170)
at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:1056)
at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:365)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:156)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:658)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:440)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
So the only solution is to reduce index size by lowering number of docs or reducing amount of indexed data per doc (which is not an option also)? We can extend RAM to 192GB, but if that helps would only be a try. The numer of "active" docs would remain at 150-160 million docs, but the number of deleted docs caused by ETL process will not be forseeable.
______________________________________________________
Gesendet mit Maills.de - mehr als nur Freemail www.maills.de
Re: Re: Config for massive inserts into Solr master
Posted by Reinhard Budenstecher <ra...@maills.de>.
> That is not correct as of version 4.0.
>
> The only kind of update I've run into that cannot proceed at the same
> time as an optimize is a deleteByQuery operation. If you do that, then
> it will block until the optimize is done, and I think it will also block
>
> any update you do after it.
>
As UPDATES, INSERTS and DELETES always contain the field ID, I can always perform an OPTIMIZE!? That sounds great. As disk is double size as index there should be enough disk space left. So I can perform a weekly OPTIMIZE on MASTER without limitations (apart from deleteByQuery) per crontab?
______________________________________________________
Gesendet mit Maills.de - mehr als nur Freemail www.maills.de
Re: Config for massive inserts into Solr master
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/12/2016 12:18 AM, Reinhard Budenstecher wrote:
> Is my assumption correct that an OPTIMIZE of index would block all
> inserts? So that all processes have to pause when I will start an
> hour-running OPTIMIZE? If so, this would also be no option for the moment.
That is not correct as of version 4.0.
The only kind of update I've run into that cannot proceed at the same
time as an optimize is a deleteByQuery operation. If you do that, then
it will block until the optimize is done, and I think it will also block
any update you do after it.
Inserts, updates, deletes by ID (uniqueKey field), and commits will all
work perfectly while an optimize is underway.
There is a workaround in USER code for the deleteByQuery problem.
Change all deleteByQuery updates in your indexing program(s) into a two
step process where you do a query with fl set to the uniqueKey field, to
gather a complete list of IDs, then do a series of delete by ID requests
with those IDs. If the query matches a large number of docs, using
cursorMark to page through them might be a good idea.
I have no idea whether the deleteByQuery problem should be considered a
bug or not, because I do not know enough about what's actually happening
when it occurs.
Thanks,
Shawn
Re: Re: Config for massive inserts into Solr master
Posted by Reinhard Budenstecher <ra...@maills.de>.
>
> That's considerably larger than you initially indicated. In just one
> index, you've got almost 300 million docs taking up well over 200GB.
> About half of them have been deleted, but they are still there. Those
> deleted docs *DO* affect operation and memory usage.
>
> Getting rid of deleted docs would go a long way towards reducing memory
>
> usage. The only effective way to get rid of them is to optimize the
> index ... but I will warn you that with an index of that size, the time
It really seems to be a matter of size :) We've extended servers RAM from 64GB to 128GB and raised heap space from 32GB to 64GB and now ETL processes are running for 3 days now without interruption. That does not satisfy me but it's a solution to keep business running for now.
Is my assumption correct that an OPTIMIZE of index would block all inserts? So that all processes have to pause when I will start an hour-running OPTIMIZE? If so, this would also be no option for the moment.
______________________________________________________
Gesendet mit Maills.de - mehr als nur Freemail www.maills.de
Re: Config for massive inserts into Solr master
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/9/2016 1:59 PM, Reinhard Budenstecher wrote:
> Solr 6.2.1 on Debian Jessie, installed with:
<snip>
> Actually, there are three cores and UI gives me following info:
> Num Docs:148652589, Max Doc:298367634, Size:219.92 GB Num
> Docs:37396140, Max Doc:38926989, Size:28.81 GB Num Docs:8601222Max
> Doc:9111004, Size:6.26 GB
That's considerably larger than you initially indicated. In just one
index, you've got almost 300 million docs taking up well over 200GB.
About half of them have been deleted, but they are still there. Those
deleted docs *DO* affect operation and memory usage.
Getting rid of deleted docs would go a long way towards reducing memory
usage. The only effective way to get rid of them is to optimize the
index ... but I will warn you that with an index of that size, the time
required for an optimize can reach into multiple hours, and will
temporarily require considerable additional disk space. The fact that
your index is on SSD will probably not improve the performance of an
optimize. When I have looked into optimize performance, it typically
proceeds at a much slower pace than most disks can sustain, because
Lucene must process the data as it rewrites it.
> Actually, Solr servers dies nearly once a day, on next shutdown, I'll
> reduce the heap size.
I don't know if that's going to help, but it might.
> How do I create such a stack trace? I have no more log informations
> than the already posted ones.
You don't need to create it. Stacktraces are logged by Solr, in a file
named solr.log, whenever most errors occur.
Thanks,
Shawn
Re: Re: Config for massive inserts into Solr master
Posted by Reinhard Budenstecher <ra...@maills.de>.
> What version of Solr? How has it been installed and started?
>
Solr 6.2.1 on Debian Jessie, installed with:
apt-get install openjdk-8-jre-headless openjdk-8-jdk-headless
wget "http://www.eu.apache.org/dist/lucene/solr/6.2.1/solr-6.2.1.tgz" && tar xvfz solr-*.tgz
./solr-*/bin/install_solr_service.sh solr-*.tgz
started with "service solr start"
> Is this a single index core with 150 million docs and 140GB index
> directory size, or is that the sum total of all the indexes on the machine?
>
Actually, there are three cores and UI gives me following info:
Num Docs:148652589, Max Doc:298367634, Size:219.92 GB
Num Docs:37396140, Max Doc:38926989, Size:28.81 GB
Num Docs:8601222Max Doc:9111004, Size:6.26 GB
but the last two cores are not important
>
> It seems unlikely to me that you would see OOM errors when indexing with
> a 32GB heap and no queries. You might try dropping the max heap to 31GB
> instead of 32GB, so your Java pointer sizes are cut in half. You might
> actually see a net increase in the amount of memory that Solr can
> utilize with that change.
Actually, Solr servers dies nearly once a day, on next shutdown, I'll reduce the heap size.
>
> Whether the errors continue or not, can you copy the full error from
> your log with stacktrace(s) so we can see it?
How do I create such a stack trace? I have no more log informations than the already posted ones.
______________________________________________________
Gesendet mit Maills.de - mehr als nur Freemail www.maills.de
Re: Config for massive inserts into Solr master
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/9/2016 12:33 PM, Reinhard Budenstecher wrote:
> We have an ETL process which updates product catalog. This produces massive inserts on MASTER, but there are no reads. Often there are thousands and hundreds of thousands of records per minute that where inserted. But sometimes I get a an OOM error, the only log entry I can find is:
>
> 2016-10-09T16:17:34.440+0200: 63872,249: [Full GC (Allocation Failure) 2016-10-09T16:17:34.440+0200: 63872,249: [CMS: 16387099K->16387087K(16777216K), 4,2227778 secs] 17782619K->17782606K(30758272K), [Metaspace: 36452K->36452K(38912K)], 4,2229287 secs] [Times: user=4,22 sys=0,01, real=4,22 secs]
>
> As I'm a bit lost with this all: is there anybody who can help me with best config for massive inserts on MASTER and massive reads on SLAVE. Is there a common approach? What details should I provide furthermore? Or is the simplest solution to raise heap on MASTER from 32GB of the available 64GB to a higher value?
What version of Solr? How has it been installed and started?
Is this a single index core with 150 million docs and 140GB index
directory size, or is that the sum total of all the indexes on the machine?
It seems unlikely to me that you would see OOM errors when indexing with
a 32GB heap and no queries. You might try dropping the max heap to 31GB
instead of 32GB, so your Java pointer sizes are cut in half. You might
actually see a net increase in the amount of memory that Solr can
utilize with that change.
Whether the errors continue or not, can you copy the full error from
your log with stacktrace(s) so we can see it?
Thanks,
Shawn