You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bruno Mannina <bm...@free.fr> on 2015/05/04 11:19:34 UTC
Delete document stop my solr 5.0 ?!
Dear Solr Users,
I have a brand new computer where I installed Ubuntu 14.04, 8Go RAM,
SOLR 5.0, Java 7
I indexed 92 000 000 docs (little text file ~2ko each)
I have around 30 fields
All work fine but each Tuesday I need to delete some docs inside, so I
create a batch file
with inside line like this:
/home/solr/solr-5.0.0/bin/post -c docdb -commit no -d
"<delete><query>f1:58644</query></delete>"
/home/solr/solr-5.0.0/bin/post -c docdb -commit no -d
"<delete><query>f1:162882</query></delete>"
......
.....
/home/solr/solr-5.0.0/bin/post -c docdb -commit yes -d
"<delete><query>f1:2868668</query></delete>"
my f1 field is my key field. It is unique.
But if my file contains more than one or two hundreds line, my solr
shutdown.
Two hundreds line shutdown always solr 5.0.
I have no error in my console, just Solr can't be reach on the port 8983.
Is exists a variable that I must increase to disable this error ?
On my old solr 3.6, I don't use the same line to delete document, I use:
java -jar -Ddata=args -Dcommit=no post.jar
"<delete><id>113422</id></delete>"
You can see that I use directly <id> not <query>, and my schema between
solr3.6 and solr5.0 is almost the same.
I have just some more fields.
why this method do not work now ?
Thanks a lot,
Bruno
---
Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active.
http://www.avast.com
Re: Delete document stop my solr 5.0 ?!
Posted by Bruno Mannina <bm...@free.fr>.
Yes it was that ! I increased the SOLR_JAVA_MEM to 2g (with 8Go Ram i do
more, 3g fail to run solr on my brand new computer)
thanks !
Le 04/05/2015 17:03, Shawn Heisey a écrit :
> On 5/4/2015 8:38 AM, Bruno Mannina wrote:
>> ok I have this OOM error in the log file ...
>>
>> #
>> # java.lang.OutOfMemoryError: Java heap space
>> # -XX:OnOutOfMemoryError="/home/solr/solr-5.0.0/bin/oom_solr.sh
>> 8983/home/solr/solr-5.0.0/server/logs"
>> # Executing /bin/sh -c "/home/solr/solr-5.0.0/bin/oom_solr.sh
>> 8983/home/solr/solr-5.0.0/server/logs"...
>> Running OOM killer script for process 28233 for Solr on port 8983
> Out Of Memory errors are a completely different problem. Solr behavior
> is completely unpredictable after an OutOfMemoryError exception, so the
> 5.0 install includes a script to run on OOME that kills Solr. It's the
> only safe way to handle that problem.
>
> Your Solr install is not being given enough Java heap memory for what it
> is being asked to do. You need to increase the heap size for Solr. If
> you look at the admin UI for Solr in a web browser, you can see what the
> max heap is set to ... on a default 5.0 install running Solr with
> "bin/solr" the max heap will be 512m ... which is VERY small. Try using
> bin/solr with the -m option, set to something like 2g (for 2 gigabytes
> of heap).
>
> Thanks,
> Shawn
>
>
>
---
Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active.
http://www.avast.com
Re: Delete document stop my solr 5.0 ?!
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/4/2015 8:38 AM, Bruno Mannina wrote:
> ok I have this OOM error in the log file ...
>
> #
> # java.lang.OutOfMemoryError: Java heap space
> # -XX:OnOutOfMemoryError="/home/solr/solr-5.0.0/bin/oom_solr.sh
> 8983/home/solr/solr-5.0.0/server/logs"
> # Executing /bin/sh -c "/home/solr/solr-5.0.0/bin/oom_solr.sh
> 8983/home/solr/solr-5.0.0/server/logs"...
> Running OOM killer script for process 28233 for Solr on port 8983
Out Of Memory errors are a completely different problem. Solr behavior
is completely unpredictable after an OutOfMemoryError exception, so the
5.0 install includes a script to run on OOME that kills Solr. It's the
only safe way to handle that problem.
Your Solr install is not being given enough Java heap memory for what it
is being asked to do. You need to increase the heap size for Solr. If
you look at the admin UI for Solr in a web browser, you can see what the
max heap is set to ... on a default 5.0 install running Solr with
"bin/solr" the max heap will be 512m ... which is VERY small. Try using
bin/solr with the -m option, set to something like 2g (for 2 gigabytes
of heap).
Thanks,
Shawn
Re: Delete document stop my solr 5.0 ?!
Posted by Bruno Mannina <bm...@free.fr>.
I increase the
formdataUploadLimitInKB
to 2048000 and the problem is the same, same error....
an idea ?
Le 04/05/2015 16:38, Bruno Mannina a écrit :
> ok I have this OOM error in the log file ...
>
> #
> # java.lang.OutOfMemoryError: Java heap space
> # -XX:OnOutOfMemoryError="/home/solr/solr-5.0.0/bin/oom_solr.sh
> 8983/home/solr/solr-5.0.0/server/logs"
> # Executing /bin/sh -c "/home/solr/solr-5.0.0/bin/oom_solr.sh
> 8983/home/solr/solr-5.0.0/server/logs"...
> Running OOM killer script for process 28233 for Solr on port 8983
> Killed process 28233
>
> I try in few minutes to increase the
>
> formdataUploadLimitInKB
>
> and I will tell you the result.
>
> Le 04/05/2015 14:58, Shawn Heisey a écrit :
>> On 5/4/2015 3:19 AM, Bruno Mannina wrote:
>>> All work fine but each Tuesday I need to delete some docs inside, so I
>>> create a batch file
>>> with inside line like this:
>>> /home/solr/solr-5.0.0/bin/post -c docdb -commit no -d
>>> "<delete><query>f1:58644</query></delete>"
>>> /home/solr/solr-5.0.0/bin/post -c docdb -commit no -d
>>> "<delete><query>f1:162882</query></delete>"
>>> ......
>>> .....
>>> /home/solr/solr-5.0.0/bin/post -c docdb -commit yes -d
>>> "<delete><query>f1:2868668</query></delete>"
>>>
>>> my f1 field is my key field. It is unique.
>>>
>>> But if my file contains more than one or two hundreds line, my solr
>>> shutdown.
>>> Two hundreds line shutdown always solr 5.0.
>>> I have no error in my console, just Solr can't be reach on the port
>>> 8983.
>>>
>>> Is exists a variable that I must increase to disable this error ?
>> As far as I know, the only limit that can affect that is the maximum
>> post size. Current versions of Solr default to a 2MB max post size,
>> using the formdataUploadLimitInKB attribute on the requestParsers
>> element in solrconfig.xml, which defaults to 2048.
>>
>> Even if that limit is exceeded by a request, it should not crash Solr,
>> it should simply log an error and ignore the request. It would be a bug
>> if Solr does crash.
>>
>> What happens if you increase that limit? Are you seeing any error
>> messages in the Solr logfile when you send that delete request?
>>
>> Thanks,
>> Shawn
>>
>>
>>
>
>
> ---
> Ce courrier électronique ne contient aucun virus ou logiciel
> malveillant parce que la protection avast! Antivirus est active.
> http://www.avast.com
>
>
>
---
Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active.
http://www.avast.com
Re: Delete document stop my solr 5.0 ?!
Posted by Bruno Mannina <bm...@free.fr>.
ok I have this OOM error in the log file ...
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="/home/solr/solr-5.0.0/bin/oom_solr.sh
8983/home/solr/solr-5.0.0/server/logs"
# Executing /bin/sh -c "/home/solr/solr-5.0.0/bin/oom_solr.sh
8983/home/solr/solr-5.0.0/server/logs"...
Running OOM killer script for process 28233 for Solr on port 8983
Killed process 28233
I try in few minutes to increase the
formdataUploadLimitInKB
and I will tell you the result.
Le 04/05/2015 14:58, Shawn Heisey a écrit :
> On 5/4/2015 3:19 AM, Bruno Mannina wrote:
>> All work fine but each Tuesday I need to delete some docs inside, so I
>> create a batch file
>> with inside line like this:
>> /home/solr/solr-5.0.0/bin/post -c docdb -commit no -d
>> "<delete><query>f1:58644</query></delete>"
>> /home/solr/solr-5.0.0/bin/post -c docdb -commit no -d
>> "<delete><query>f1:162882</query></delete>"
>> ......
>> .....
>> /home/solr/solr-5.0.0/bin/post -c docdb -commit yes -d
>> "<delete><query>f1:2868668</query></delete>"
>>
>> my f1 field is my key field. It is unique.
>>
>> But if my file contains more than one or two hundreds line, my solr
>> shutdown.
>> Two hundreds line shutdown always solr 5.0.
>> I have no error in my console, just Solr can't be reach on the port 8983.
>>
>> Is exists a variable that I must increase to disable this error ?
> As far as I know, the only limit that can affect that is the maximum
> post size. Current versions of Solr default to a 2MB max post size,
> using the formdataUploadLimitInKB attribute on the requestParsers
> element in solrconfig.xml, which defaults to 2048.
>
> Even if that limit is exceeded by a request, it should not crash Solr,
> it should simply log an error and ignore the request. It would be a bug
> if Solr does crash.
>
> What happens if you increase that limit? Are you seeing any error
> messages in the Solr logfile when you send that delete request?
>
> Thanks,
> Shawn
>
>
>
---
Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active.
http://www.avast.com
Re: Delete document stop my solr 5.0 ?!
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/4/2015 3:19 AM, Bruno Mannina wrote:
> All work fine but each Tuesday I need to delete some docs inside, so I
> create a batch file
> with inside line like this:
> /home/solr/solr-5.0.0/bin/post -c docdb -commit no -d
> "<delete><query>f1:58644</query></delete>"
> /home/solr/solr-5.0.0/bin/post -c docdb -commit no -d
> "<delete><query>f1:162882</query></delete>"
> ......
> .....
> /home/solr/solr-5.0.0/bin/post -c docdb -commit yes -d
> "<delete><query>f1:2868668</query></delete>"
>
> my f1 field is my key field. It is unique.
>
> But if my file contains more than one or two hundreds line, my solr
> shutdown.
> Two hundreds line shutdown always solr 5.0.
> I have no error in my console, just Solr can't be reach on the port 8983.
>
> Is exists a variable that I must increase to disable this error ?
As far as I know, the only limit that can affect that is the maximum
post size. Current versions of Solr default to a 2MB max post size,
using the formdataUploadLimitInKB attribute on the requestParsers
element in solrconfig.xml, which defaults to 2048.
Even if that limit is exceeded by a request, it should not crash Solr,
it should simply log an error and ignore the request. It would be a bug
if Solr does crash.
What happens if you increase that limit? Are you seeing any error
messages in the Solr logfile when you send that delete request?
Thanks,
Shawn
Re: Delete document stop my solr 5.0 ?!
Posted by Chris Hostetter <ho...@fucit.org>.
XY-ish problem -- if you are deleting a bunch of documents by id, why have
you switched from using delete-by-id to using delete-by-query? What drove
that decision? Did you try using delete-by-query in your 3.6 setup?
: my f1 field is my key field. It is unique.
...
: On my old solr 3.6, I don't use the same line to delete document, I use:
: java -jar -Ddata=args -Dcommit=no post.jar
: "<delete><id>113422</id></delete>"
...
: why this method do not work now ?
https://people.apache.org/~hossman/#xyproblem
XY Problem
Your question appears to be an "XY Problem" ... that is: you are dealing
with "X", you are assuming "Y" will help you, and you are asking about "Y"
without giving more details about the "X" so that we can understand the
full issue. Perhaps the best solution doesn't involve "Y" at all?
See Also: http://www.perlmonks.org/index.pl?node_id=542341
-Hoss
http://www.lucidworks.com/