You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Li Li <fa...@gmail.com> on 2010/07/07 04:37:26 UTC
index format error because disk full
the index file is ill-formated because disk full when feeding. Can I
roll back to last version? Is there any method to avoid unexpected
errors when indexing? attachments are my segment_N
Re: index format error because disk full
Posted by jg lin <li...@gmail.com>.
老子喜欢用汉语，竟敢骂我，这就是洋人的素质，鄙视。有些人就是自卑到骨子里了。。。
2010/7/7 <ba...@gmail.com>
> poopy
> Sent via BlackBerry from T-Mobile
>
> -----Original Message-----
> From: Li Li <fa...@gmail.com>
> Date: Wed, 7 Jul 2010 10:48:53
> To: <ja...@lucene.apache.org>
> Reply-To: java-user@lucene.apache.org
> Subject: Re: index format error because disk full
>
> 会
> 在 2010年7月7日 上午10:46，jg lin <li...@gmail.com> 写道：
> > 你会说汉语吗(⊙_⊙)？
> >
> > 2010/7/7 Li Li <fa...@gmail.com>
> >
> >> ---------- Forwarded message ----------
> >> From: Li Li <fa...@gmail.com>
> >> Date: 2010/7/7
> >> Subject: index format error because disk full
> >> To: solr-user@lucene.apache.org
> >>
> >>
> >> the index file is ill-formated because disk full when feeding. Can I
> >> roll back to last version? Is there any method to avoid unexpected
> >> errors when indexing? attachments are my segment_N
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: index format error because disk full
Posted by ba...@gmail.com.
poopy
Sent via BlackBerry from T-Mobile
-----Original Message-----
From: Li Li <fa...@gmail.com>
Date: Wed, 7 Jul 2010 10:48:53
To: <ja...@lucene.apache.org>
Reply-To: java-user@lucene.apache.org
Subject: Re: index format error because disk full
会
在 2010年7月7日 上午10:46，jg lin <li...@gmail.com> 写道：
> 你会说汉语吗(⊙_⊙)？
>
> 2010/7/7 Li Li <fa...@gmail.com>
>
>> ---------- Forwarded message ----------
>> From: Li Li <fa...@gmail.com>
>> Date: 2010/7/7
>> Subject: index format error because disk full
>> To: solr-user@lucene.apache.org
>>
>>
>> the index file is ill-formated because disk full when feeding. Can I
>> roll back to last version? Is there any method to avoid unexpected
>> errors when indexing? attachments are my segment_N
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: index format error because disk full
Posted by luby <zh...@neusoft.com>.
�������⣬�������������������ˡ�����ôһ��ȫ���綼֪���ˡ�
�� 2010-7-7 11:03, Li Li �:
> лл
>
> �� 2010��7��7�� ����10:53��jg lin <li...@gmail.com> ���
>
>> �Ӹ�QQȺ����18038594����������Ҳ��ᡣ
>>
>> Li Li <fa...@gmail.com> � 2010��7��7������10:48 ������
>>
>>
>>> ��
>>> �� 2010��7��7�� ����10:46��jg lin <li...@gmail.com> ���
>>> > ���˵������(��_��)��
>>>
>>>> 2010/7/7 Li Li <fa...@gmail.com>
>>>>
>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: Li Li <fa...@gmail.com>
>>>>> Date: 2010/7/7
>>>>> Subject: index format error because disk full
>>>>> To: solr-user@lucene.apache.org
>>>>>
>>>>>
>>>>> the index file is ill-formated because disk full when feeding. Can I
>>>>> roll back to last version? Is there any method to avoid unexpected
>>>>> errors when indexing? attachments are my segment_N
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
Re: index format error because disk full
Posted by Li Li <fa...@gmail.com>.
лл
�� 2010��7��7�� ����10:53��jg lin <li...@gmail.com> ���
> �Ӹ�QQȺ����18038594����������Ҳ��ᡣ
>
> Li Li <fa...@gmail.com> � 2010��7��7������10:48 ������
>
>> ��
>> �� 2010��7��7�� ����10:46��jg lin <li...@gmail.com> ���
>> > ���˵������(��_��)��
>> >
>> > 2010/7/7 Li Li <fa...@gmail.com>
>> >
>> >> ---------- Forwarded message ----------
>> >> From: Li Li <fa...@gmail.com>
>> >> Date: 2010/7/7
>> >> Subject: index format error because disk full
>> >> To: solr-user@lucene.apache.org
>> >>
>> >>
>> >> the index file is ill-formated because disk full when feeding. Can I
>> >> roll back to last version? Is there any method to avoid unexpected
>> >> errors when indexing? attachments are my segment_N
>> >>
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: index format error because disk full
Posted by jg lin <li...@gmail.com>.
�Ӹ�QQȺ����18038594����������Ҳ��ᡣ
Li Li <fa...@gmail.com> � 2010��7��7������10:48 ������
> ��
> �� 2010��7��7�� ����10:46��jg lin <li...@gmail.com> ���
> > ���˵������(��_��)��
> >
> > 2010/7/7 Li Li <fa...@gmail.com>
> >
> >> ---------- Forwarded message ----------
> >> From: Li Li <fa...@gmail.com>
> >> Date: 2010/7/7
> >> Subject: index format error because disk full
> >> To: solr-user@lucene.apache.org
> >>
> >>
> >> the index file is ill-formated because disk full when feeding. Can I
> >> roll back to last version? Is there any method to avoid unexpected
> >> errors when indexing? attachments are my segment_N
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: index format error because disk full
Posted by Li Li <fa...@gmail.com>.
会
在 2010年7月7日 上午10:46，jg lin <li...@gmail.com> 写道：
> 你会说汉语吗(⊙_⊙)？
>
> 2010/7/7 Li Li <fa...@gmail.com>
>
>> ---------- Forwarded message ----------
>> From: Li Li <fa...@gmail.com>
>> Date: 2010/7/7
>> Subject: index format error because disk full
>> To: solr-user@lucene.apache.org
>>
>>
>> the index file is ill-formated because disk full when feeding. Can I
>> roll back to last version? Is there any method to avoid unexpected
>> errors when indexing? attachments are my segment_N
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: index format error because disk full
Posted by jg lin <li...@gmail.com>.
你会说汉语吗(⊙_⊙)？
2010/7/7 Li Li <fa...@gmail.com>
> ---------- Forwarded message ----------
> From: Li Li <fa...@gmail.com>
> Date: 2010/7/7
> Subject: index format error because disk full
> To: solr-user@lucene.apache.org
>
>
> the index file is ill-formated because disk full when feeding. Can I
> roll back to last version? Is there any method to avoid unexpected
> errors when indexing? attachments are my segment_N
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Fwd: index format error because disk full
Posted by Li Li <fa...@gmail.com>.
---------- Forwarded message ----------
From: Li Li <fa...@gmail.com>
Date: 2010/7/7
Subject: index format error because disk full
To: solr-user@lucene.apache.org
the index file is ill-formated because disk full when feeding. Can I
roll back to last version? Is there any method to avoid unexpected
errors when indexing? attachments are my segment_N
Re: index format error because disk full
Posted by Michael McCandless <lu...@mikemccandless.com>.
Disk full should never lead to index corruption (except for very old
versions of Lucene).
Lucene always writes (and closes) all files associated with the
segment, then fsync's them, before writing & fsync'ing the segments_N
file that refers to these files.
Can you describe in more detail the events that led up to the
zero-bytes del file? What OS/filesystem?
Is there any external process that could have truncated the file? Or
possibly filesystem corruption?
Mike
On Wed, Jul 7, 2010 at 10:12 PM, Li Li <fa...@gmail.com> wrote:
> I use SegmentInfos to read the segment_N file and found the error is
> that it try to load deletedDocs but the .del file's size is 0(because
> of disk error) . So I use SegmentInfos to set delGen=-1 to ignore
> deleted Docs.
> But I think there is some bug. The logic of write my be -- it first
> writes the .del file then write the segment_N file. But it only write
> to buffer and don't flush to disk immediately. So when disk full. it
> may happen that segment_N file is flushed but del file faild.
>
> 2010/7/8 Lance Norskog <go...@gmail.com>:
>> If autocommit does not to an automatic rollback, that is a serious bug.
>>
>> There should be a way to detect that an automatic rollback has
>> happened, but I don't know what it is. Maybe something in the Solr
>> MBeans?
>>
>> On Wed, Jul 7, 2010 at 5:41 AM, osocurious2 <ke...@realestate.com> wrote:
>>>
>>> I haven't used this myself, but Solr supports a
>>> http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22 rollback
>>> function. It is supposed to rollback to the state at the previous commit. So
>>> you may want to turn off auto-commit on the index you are updating if you
>>> want to control what that last commit level is.
>>>
>>> However, in your case if the index gets corrupted due to a disk full
>>> situation, I don't know what rollback would do, if anything, to help. You
>>> may need to play with the scenario to see what would happen.
>>>
>>> If you are using the DataImportHandler it may handle the rollback for
>>> you...again, however, it may not deal with disk full situations gracefully
>>> either.
>>> --
>>> View this message in context: http://lucene.472066.n3.nabble.com/index-format-error-because-disk-full-tp948249p948968.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
>
Re: index format error because disk full
Posted by Li Li <fa...@gmail.com>.
I use SegmentInfos to read the segment_N file and found the error is
that it try to load deletedDocs but the .del file's size is 0(because
of disk error) . So I use SegmentInfos to set delGen=-1 to ignore
deleted Docs.
But I think there is some bug. The logic of write my be -- it first
writes the .del file then write the segment_N file. But it only write
to buffer and don't flush to disk immediately. So when disk full. it
may happen that segment_N file is flushed but del file faild.
2010/7/8 Lance Norskog <go...@gmail.com>:
> If autocommit does not to an automatic rollback, that is a serious bug.
>
> There should be a way to detect that an automatic rollback has
> happened, but I don't know what it is. Maybe something in the Solr
> MBeans?
>
> On Wed, Jul 7, 2010 at 5:41 AM, osocurious2 <ke...@realestate.com> wrote:
>>
>> I haven't used this myself, but Solr supports a
>> http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22 rollback
>> function. It is supposed to rollback to the state at the previous commit. So
>> you may want to turn off auto-commit on the index you are updating if you
>> want to control what that last commit level is.
>>
>> However, in your case if the index gets corrupted due to a disk full
>> situation, I don't know what rollback would do, if anything, to help. You
>> may need to play with the scenario to see what would happen.
>>
>> If you are using the DataImportHandler it may handle the rollback for
>> you...again, however, it may not deal with disk full situations gracefully
>> either.
>> --
>> View this message in context: http://lucene.472066.n3.nabble.com/index-format-error-because-disk-full-tp948249p948968.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: index format error because disk full
Posted by Lance Norskog <go...@gmail.com>.
If autocommit does not to an automatic rollback, that is a serious bug.
There should be a way to detect that an automatic rollback has
happened, but I don't know what it is. Maybe something in the Solr
MBeans?
On Wed, Jul 7, 2010 at 5:41 AM, osocurious2 <ke...@realestate.com> wrote:
>
> I haven't used this myself, but Solr supports a
> http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22 rollback
> function. It is supposed to rollback to the state at the previous commit. So
> you may want to turn off auto-commit on the index you are updating if you
> want to control what that last commit level is.
>
> However, in your case if the index gets corrupted due to a disk full
> situation, I don't know what rollback would do, if anything, to help. You
> may need to play with the scenario to see what would happen.
>
> If you are using the DataImportHandler it may handle the rollback for
> you...again, however, it may not deal with disk full situations gracefully
> either.
> --
> View this message in context: http://lucene.472066.n3.nabble.com/index-format-error-because-disk-full-tp948249p948968.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Lance Norskog
goksron@gmail.com
Re: index format error because disk full
Posted by osocurious2 <ke...@realestate.com>.
I haven't used this myself, but Solr supports a
http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22 rollback
function. It is supposed to rollback to the state at the previous commit. So
you may want to turn off auto-commit on the index you are updating if you
want to control what that last commit level is.
However, in your case if the index gets corrupted due to a disk full
situation, I don't know what rollback would do, if anything, to help. You
may need to play with the scenario to see what would happen.
If you are using the DataImportHandler it may handle the rollback for
you...again, however, it may not deal with disk full situations gracefully
either.
--
View this message in context: http://lucene.472066.n3.nabble.com/index-format-error-because-disk-full-tp948249p948968.html
Sent from the Solr - User mailing list archive at Nabble.com.