You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Gavin Harcourt <gh...@brainboxdigital.com> on 2016/05/31 12:41:22 UTC
Solr leaking references to deleted files
Hi All,
I've noticed on some of my solr nodes that the disk usage is increasing
over time. After checking the output of lsof I found hundreds of
references to deleted index files being held by solr. This totaled 24GB
on a 16GB index. A restart of solr can obviously fix this but this is
not an ideal solution. We are running solr 5.4.0 on OpenJDK 1.8.0_91.
We are using the Concurrent Mark Sweep GC although I've also seen the
same problem on nodes using the G1 GC. Our update handler has autoCommit
and softAutoCommit enabled (at different intervals). We are using solr
cloud and have multiple shards with 2 nodes each in our collections.
I've not seen any pattern between this appearing on leaders or replicas.
Not all my nodes appear to be exhibiting the problem either. Our usage
pattern does involve a lot of churn in our index with the majority of
documents being updated/deleted every day.
Searching JIRA and the web in general I could only find references to
this sort of problem when running solr in tomcat. Can anyone suggest a
reason why this might be happening or a way I can manage it without
needing to restart solr?
Example lsof output:
java 1100 s123 DEL REG 202,3 8919406
/home/s123/solr/data/uk_shard2_replica1/data/index/_3m9s.fdt
java 1100 s123 DEL REG 202,3 8919159
/home/s123/solr/data/uk_shard2_replica1/data/index/_3mnk.tvd
java 1100 s123 DEL REG 202,3 8919150
/home/s123/solr/data/uk_shard2_replica1/data/index/_3mnk_Lucene50_0.tim
java 1100 s123 DEL REG 202,3 8919094
/home/s123/solr/data/uk_shard2_replica1/data/index/_3mq1_Lucene50_0.tim
java 1100 s123 DEL REG 202,3 8919103
/home/s123/solr/data/uk_shard2_replica1/data/index/_3mq1.tvd
Regards,
Gavin.
Re: Solr leaking references to deleted files
Posted by Erick Erickson <er...@gmail.com>.
Cool, please let us know what you find out.
On Tue, May 31, 2016 at 8:34 AM, Gavin Harcourt
<gh...@brainboxdigital.com> wrote:
> Those two bugs would make sense as we have been reloading the cores quite
> frequently recently to apply new config and schema changes. I'll keep an eye
> on the situation now our reload spree has ended and see if it recurs.
>
> Thanks,
> Gavin.
>
> On 31/05/16 16:14, Erick Erickson wrote:
>>
>> Possibly:
>> SOLR-9116 or SOLR-9117? Note those two require that the core be
>> reloaded, so you have to be doing something a bit unusual for them to
>> be the problem.
>>
>> Best,
>> Erick
>>
>> On Tue, May 31, 2016 at 5:41 AM, Gavin Harcourt
>> <gh...@brainboxdigital.com> wrote:
>>>
>>> Hi All,
>>>
>>> I've noticed on some of my solr nodes that the disk usage is increasing
>>> over
>>> time. After checking the output of lsof I found hundreds of references to
>>> deleted index files being held by solr. This totaled 24GB on a 16GB
>>> index. A
>>> restart of solr can obviously fix this but this is not an ideal solution.
>>> We
>>> are running solr 5.4.0 on OpenJDK 1.8.0_91. We are using the Concurrent
>>> Mark Sweep GC although I've also seen the same problem on nodes using the
>>> G1
>>> GC. Our update handler has autoCommit and softAutoCommit enabled (at
>>> different intervals). We are using solr cloud and have multiple shards
>>> with
>>> 2 nodes each in our collections. I've not seen any pattern between this
>>> appearing on leaders or replicas. Not all my nodes appear to be
>>> exhibiting
>>> the problem either. Our usage pattern does involve a lot of churn in our
>>> index with the majority of documents being updated/deleted every day.
>>>
>>> Searching JIRA and the web in general I could only find references to
>>> this
>>> sort of problem when running solr in tomcat. Can anyone suggest a reason
>>> why
>>> this might be happening or a way I can manage it without needing to
>>> restart
>>> solr?
>>>
>>> Example lsof output:
>>> java 1100 s123 DEL REG 202,3 8919406
>>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3m9s.fdt
>>> java 1100 s123 DEL REG 202,3 8919159
>>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mnk.tvd
>>> java 1100 s123 DEL REG 202,3 8919150
>>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mnk_Lucene50_0.tim
>>> java 1100 s123 DEL REG 202,3 8919094
>>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mq1_Lucene50_0.tim
>>> java 1100 s123 DEL REG 202,3 8919103
>>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mq1.tvd
>>>
>>> Regards,
>>> Gavin.
>>>
>
Re: Solr leaking references to deleted files
Posted by Gavin Harcourt <gh...@brainboxdigital.com>.
Those two bugs would make sense as we have been reloading the cores
quite frequently recently to apply new config and schema changes. I'll
keep an eye on the situation now our reload spree has ended and see if
it recurs.
Thanks,
Gavin.
On 31/05/16 16:14, Erick Erickson wrote:
> Possibly:
> SOLR-9116 or SOLR-9117? Note those two require that the core be
> reloaded, so you have to be doing something a bit unusual for them to
> be the problem.
>
> Best,
> Erick
>
> On Tue, May 31, 2016 at 5:41 AM, Gavin Harcourt
> <gh...@brainboxdigital.com> wrote:
>> Hi All,
>>
>> I've noticed on some of my solr nodes that the disk usage is increasing over
>> time. After checking the output of lsof I found hundreds of references to
>> deleted index files being held by solr. This totaled 24GB on a 16GB index. A
>> restart of solr can obviously fix this but this is not an ideal solution. We
>> are running solr 5.4.0 on OpenJDK 1.8.0_91. We are using the Concurrent
>> Mark Sweep GC although I've also seen the same problem on nodes using the G1
>> GC. Our update handler has autoCommit and softAutoCommit enabled (at
>> different intervals). We are using solr cloud and have multiple shards with
>> 2 nodes each in our collections. I've not seen any pattern between this
>> appearing on leaders or replicas. Not all my nodes appear to be exhibiting
>> the problem either. Our usage pattern does involve a lot of churn in our
>> index with the majority of documents being updated/deleted every day.
>>
>> Searching JIRA and the web in general I could only find references to this
>> sort of problem when running solr in tomcat. Can anyone suggest a reason why
>> this might be happening or a way I can manage it without needing to restart
>> solr?
>>
>> Example lsof output:
>> java 1100 s123 DEL REG 202,3 8919406
>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3m9s.fdt
>> java 1100 s123 DEL REG 202,3 8919159
>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mnk.tvd
>> java 1100 s123 DEL REG 202,3 8919150
>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mnk_Lucene50_0.tim
>> java 1100 s123 DEL REG 202,3 8919094
>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mq1_Lucene50_0.tim
>> java 1100 s123 DEL REG 202,3 8919103
>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mq1.tvd
>>
>> Regards,
>> Gavin.
>>
Re: Solr leaking references to deleted files
Posted by Erick Erickson <er...@gmail.com>.
Possibly:
SOLR-9116 or SOLR-9117? Note those two require that the core be
reloaded, so you have to be doing something a bit unusual for them to
be the problem.
Best,
Erick
On Tue, May 31, 2016 at 5:41 AM, Gavin Harcourt
<gh...@brainboxdigital.com> wrote:
> Hi All,
>
> I've noticed on some of my solr nodes that the disk usage is increasing over
> time. After checking the output of lsof I found hundreds of references to
> deleted index files being held by solr. This totaled 24GB on a 16GB index. A
> restart of solr can obviously fix this but this is not an ideal solution. We
> are running solr 5.4.0 on OpenJDK 1.8.0_91. We are using the Concurrent
> Mark Sweep GC although I've also seen the same problem on nodes using the G1
> GC. Our update handler has autoCommit and softAutoCommit enabled (at
> different intervals). We are using solr cloud and have multiple shards with
> 2 nodes each in our collections. I've not seen any pattern between this
> appearing on leaders or replicas. Not all my nodes appear to be exhibiting
> the problem either. Our usage pattern does involve a lot of churn in our
> index with the majority of documents being updated/deleted every day.
>
> Searching JIRA and the web in general I could only find references to this
> sort of problem when running solr in tomcat. Can anyone suggest a reason why
> this might be happening or a way I can manage it without needing to restart
> solr?
>
> Example lsof output:
> java 1100 s123 DEL REG 202,3 8919406
> /home/s123/solr/data/uk_shard2_replica1/data/index/_3m9s.fdt
> java 1100 s123 DEL REG 202,3 8919159
> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mnk.tvd
> java 1100 s123 DEL REG 202,3 8919150
> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mnk_Lucene50_0.tim
> java 1100 s123 DEL REG 202,3 8919094
> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mq1_Lucene50_0.tim
> java 1100 s123 DEL REG 202,3 8919103
> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mq1.tvd
>
> Regards,
> Gavin.
>