You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by sunnyfr <jo...@gmail.com> on 2009/03/19 16:12:53 UTC
Problem for replication : segment optimized automaticly
Hi everybody ... still me :)
hoo happy day :)
Just, I dont get where I miss something, I will try to be clear.
this is my index folder (and we can notice the evolution according to the
delta import every 30mn) :
root@search-01:/data/solr# ls video/data/index/
_2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx
_2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx _2ben.frq
_2ben.prx _2ben.tis _2beo.fdx segments.gen
_2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del
_2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm _2ben.nrm
_2ben.tii _2beo.fdt _2beo.fnm segments_230x
root@search-01:/data/solr# ls video/data/index/
_2bel.fdt _2bel.frq _2bel.tii _2bel.tvf _2bem.fdt _2bem.frq
_2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm _2beo.tis
_2beo.tvx
_2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx _2bem.fdx _2bem.nrm
_2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx _2beo.tvd
segments.gen
_2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx
_2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii _2beo.tvf
segments_230x
root@search-01:/data/solr# ls video/data/index/
_2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx _2beo.tii
_2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen segments_230y
So as you can notice my segments increased which is perfect for my
replication (faster to get JUST last segments)
But like you can see in my last ls my segment has been optimized.
Like I can notice in my log :
Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import
completed successfully without optimization
Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
org.apache.solr.update.DirectUpdateHandler2 commit INFO: start
commit(optimize=true,waitFlush=false,waitSearcher=true)
But I didn't fire any optimize, my delta-import is fired like :
/solr/video/dataimport?command=delta-import&optimize=false
Solrconfig.xml : autocommit turnd off
<!--<luceneAutoCommit>false</luceneAutoCommit>-->
<!-- <commitLockTimeout>18500</commitLockTimeout>-->
<!-- autocommit pending docs if certain criteria are met
<autoCommit>
<maxDocs>10000</maxDocs>
</autoCommit>
-->
Maybe it comes from lucene parameters?
<!-- options specific to the main on-disk lucene index -->
<useCompoundFile>false</useCompoundFile>
<ramBufferSizeMB>50</ramBufferSizeMB>
<mergeFactor>50</mergeFactor>
<!-- Deprecated -->
<!--<maxBufferedDocs>1000</maxBufferedDocs>-->
<maxMergeDocs>2147483647</maxMergeDocs>
<maxFieldLength>10000</maxFieldLength>
Thanks a lot for your help,
Sunny
--
View this message in context: http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22601442.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem for replication : segment optimized automaticly
Posted by sunnyfr <jo...@gmail.com>.
Where do you see that I can maybe check in my code, is it in updateHandler2 ?
thanks,
Shalin Shekhar Mangar wrote:
>
> Lucene will automatically merge segments when they exceed the mergeFactor.
> This may be one reason but I'm not sure.
>
> I checked DataImportHandler's code again. It won't optimize if
> optimize=false is specified.
>
> On Mon, Mar 23, 2009 at 12:43 AM, sunnyfr <jo...@gmail.com> wrote:
>
>>
>> Do you have any idea ???
>> :(
>>
>> cheer,
>>
>>
>> sunnyfr wrote:
>> >
>> > Hi everybody ... still me :)
>> > hoo happy day :)
>> >
>> > Just, I dont get where I miss something, I will try to be clear.
>> >
>> > this is my index folder (and we can notice the evolution according to
>> the
>> > delta import every 30mn) :
>> >
>> > root@search-01:/data/solr# ls video/data/index/
>> > _2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx
>> > _2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx
>> > _2ben.frq _2ben.prx _2ben.tis _2beo.fdx segments.gen
>> > _2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del
>> > _2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm
>> > _2ben.nrm _2ben.tii _2beo.fdt _2beo.fnm segments_230x
>> > root@search-01:/data/solr# ls video/data/index/
>> > _2bel.fdt _2bel.frq _2bel.tii _2bel.tvf _2bem.fdt _2bem.frq
>> > _2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm
>> > _2beo.tis _2beo.tvx
>> > _2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx _2bem.fdx _2bem.nrm
>> > _2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx
>> > _2beo.tvd segments.gen
>> > _2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx
>> > _2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii
>> > _2beo.tvf segments_230x
>> > root@search-01:/data/solr# ls video/data/index/
>> > _2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx
>> > _2beo.tii _2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen
>> > segments_230y
>> >
>> > So as you can notice my segments increased which is perfect for my
>> > replication (faster to get JUST last segments)
>> > But like you can see in my last ls my segment has been optimized.
>> >
>> > Like I can notice in my log :
>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
>> > org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import
>> > completed successfully without optimization
>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
>> > org.apache.solr.update.DirectUpdateHandler2 commit INFO: start
>> > commit(optimize=true,waitFlush=false,waitSearcher=true)
>> >
>> > But I didn't fire any optimize, my delta-import is fired like :
>> > /solr/video/dataimport?command=delta-import&optimize=false
>> >
>> > Solrconfig.xml : autocommit turnd off
>> > <!--<luceneAutoCommit>false</luceneAutoCommit>-->
>> > <!-- <commitLockTimeout>18500</commitLockTimeout>-->
>> > <!-- autocommit pending docs if certain criteria are met
>> > <autoCommit>
>> > <maxDocs>10000</maxDocs>
>> > </autoCommit>
>> > -->
>> >
>> > Maybe it comes from lucene parameters?
>> > <!-- options specific to the main on-disk lucene index -->
>> > <useCompoundFile>false</useCompoundFile>
>> > <ramBufferSizeMB>50</ramBufferSizeMB>
>> > <mergeFactor>50</mergeFactor>
>> > <!-- Deprecated -->
>> > <!--<maxBufferedDocs>1000</maxBufferedDocs>-->
>> > <maxMergeDocs>2147483647</maxMergeDocs>
>> > <maxFieldLength>10000</maxFieldLength>
>> >
>> > Thanks a lot for your help,
>> > Sunny
>> >
>> >
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22649412.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22661653.html
Sent from the Solr - User mailing list archive at Nabble.com.
Weirdness in 1052
Posted by Vernon Chapman <ch...@gmail.com>.
Hello All,
I checked out a fresh version of 1052
I copied the database.example and solr.example to database.yml & solr.yml
I have a multicore external solr instance and so I changed the urls in
the solr.yml file to reflect that.
I have changed all the url to http://localhost:8080/solr/content.
If I start BL using apache2+mod_rails or using ./script/server I get :
private method `gsub' called for #<Curl::Err::ConnectionFailedError:
couldn't connect to server>
The funny thing is that I can connect to the jetty server on 8080 from
my browser. In an effort to track it down, I ran
grep -r 8983 . in the rails directory and came up with 3 lines from
the ./vendor/plugins/blacklight/install/solr.yml file.
I am about to start a broject and I wanted to use the demo app as a
starting point but I can't if I can't connect. Any Ideas?
Thanks
Vernon
a.k.a - g8tor
Re: Problem for replication : segment optimized automaticly
Posted by sunnyfr <jo...@gmail.com>.
Obviously mine yes ... :s
Shalin Shekhar Mangar wrote:
>
> Lucene will automatically merge segments when they exceed the mergeFactor.
> This may be one reason but I'm not sure.
>
> I checked DataImportHandler's code again. It won't optimize if
> optimize=false is specified.
>
> On Mon, Mar 23, 2009 at 12:43 AM, sunnyfr <jo...@gmail.com> wrote:
>
>>
>> Do you have any idea ???
>> :(
>>
>> cheer,
>>
>>
>> sunnyfr wrote:
>> >
>> > Hi everybody ... still me :)
>> > hoo happy day :)
>> >
>> > Just, I dont get where I miss something, I will try to be clear.
>> >
>> > this is my index folder (and we can notice the evolution according to
>> the
>> > delta import every 30mn) :
>> >
>> > root@search-01:/data/solr# ls video/data/index/
>> > _2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx
>> > _2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx
>> > _2ben.frq _2ben.prx _2ben.tis _2beo.fdx segments.gen
>> > _2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del
>> > _2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm
>> > _2ben.nrm _2ben.tii _2beo.fdt _2beo.fnm segments_230x
>> > root@search-01:/data/solr# ls video/data/index/
>> > _2bel.fdt _2bel.frq _2bel.tii _2bel.tvf _2bem.fdt _2bem.frq
>> > _2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm
>> > _2beo.tis _2beo.tvx
>> > _2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx _2bem.fdx _2bem.nrm
>> > _2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx
>> > _2beo.tvd segments.gen
>> > _2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx
>> > _2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii
>> > _2beo.tvf segments_230x
>> > root@search-01:/data/solr# ls video/data/index/
>> > _2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx
>> > _2beo.tii _2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen
>> > segments_230y
>> >
>> > So as you can notice my segments increased which is perfect for my
>> > replication (faster to get JUST last segments)
>> > But like you can see in my last ls my segment has been optimized.
>> >
>> > Like I can notice in my log :
>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
>> > org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import
>> > completed successfully without optimization
>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
>> > org.apache.solr.update.DirectUpdateHandler2 commit INFO: start
>> > commit(optimize=true,waitFlush=false,waitSearcher=true)
>> >
>> > But I didn't fire any optimize, my delta-import is fired like :
>> > /solr/video/dataimport?command=delta-import&optimize=false
>> >
>> > Solrconfig.xml : autocommit turnd off
>> > <!--<luceneAutoCommit>false</luceneAutoCommit>-->
>> > <!-- <commitLockTimeout>18500</commitLockTimeout>-->
>> > <!-- autocommit pending docs if certain criteria are met
>> > <autoCommit>
>> > <maxDocs>10000</maxDocs>
>> > </autoCommit>
>> > -->
>> >
>> > Maybe it comes from lucene parameters?
>> > <!-- options specific to the main on-disk lucene index -->
>> > <useCompoundFile>false</useCompoundFile>
>> > <ramBufferSizeMB>50</ramBufferSizeMB>
>> > <mergeFactor>50</mergeFactor>
>> > <!-- Deprecated -->
>> > <!--<maxBufferedDocs>1000</maxBufferedDocs>-->
>> > <maxMergeDocs>2147483647</maxMergeDocs>
>> > <maxFieldLength>10000</maxFieldLength>
>> >
>> > Thanks a lot for your help,
>> > Sunny
>> >
>> >
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22649412.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22661570.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem for replication : segment optimized automaticly
Posted by sunnyfr <jo...@gmail.com>.
How can I stop this?
Noble Paul നോബിള് नोब्ळ् wrote:
>
> if the DIH status does not say that it optimized, it is lucene
> mergeing the segments
>
> On Mon, Mar 23, 2009 at 8:15 PM, sunnyfr <jo...@gmail.com> wrote:
>>
>> I checked this out but It doesn't say nothing about optimizing.
>> I'm sure it's lucene part about merging or I don't know ...??
>>
>>
>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>
>>> the easiest way to find out what DIH did is to hit it's status
>>> command. It will give you a brief description of what all it did
>>> during last import
>>>
>>> On Mon, Mar 23, 2009 at 12:59 AM, Shalin Shekhar Mangar
>>> <sh...@gmail.com> wrote:
>>>> Lucene will automatically merge segments when they exceed the
>>>> mergeFactor.
>>>> This may be one reason but I'm not sure.
>>>>
>>>> I checked DataImportHandler's code again. It won't optimize if
>>>> optimize=false is specified.
>>>>
>>>> On Mon, Mar 23, 2009 at 12:43 AM, sunnyfr <jo...@gmail.com> wrote:
>>>>
>>>>>
>>>>> Do you have any idea ???
>>>>> :(
>>>>>
>>>>> cheer,
>>>>>
>>>>>
>>>>> sunnyfr wrote:
>>>>> >
>>>>> > Hi everybody ... still me :)
>>>>> > hoo happy day :)
>>>>> >
>>>>> > Just, I dont get where I miss something, I will try to be clear.
>>>>> >
>>>>> > this is my index folder (and we can notice the evolution according
>>>>> to
>>>>> the
>>>>> > delta import every 30mn) :
>>>>> >
>>>>> > root@search-01:/data/solr# ls video/data/index/
>>>>> > _2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx
>>>>> > _2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx
>>>>> > _2ben.frq _2ben.prx _2ben.tis _2beo.fdx segments.gen
>>>>> > _2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del
>>>>> > _2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm
>>>>> > _2ben.nrm _2ben.tii _2beo.fdt _2beo.fnm segments_230x
>>>>> > root@search-01:/data/solr# ls video/data/index/
>>>>> > _2bel.fdt _2bel.frq _2bel.tii _2bel.tvf _2bem.fdt _2bem.frq
>>>>> > _2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm
>>>>> > _2beo.tis _2beo.tvx
>>>>> > _2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx _2bem.fdx _2bem.nrm
>>>>> > _2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx
>>>>> > _2beo.tvd segments.gen
>>>>> > _2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx
>>>>> > _2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii
>>>>> > _2beo.tvf segments_230x
>>>>> > root@search-01:/data/solr# ls video/data/index/
>>>>> > _2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx
>>>>> > _2beo.tii _2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen
>>>>> > segments_230y
>>>>> >
>>>>> > So as you can notice my segments increased which is perfect for my
>>>>> > replication (faster to get JUST last segments)
>>>>> > But like you can see in my last ls my segment has been optimized.
>>>>> >
>>>>> > Like I can notice in my log :
>>>>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
>>>>> > org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full
>>>>> Import
>>>>> > completed successfully without optimization
>>>>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
>>>>> > org.apache.solr.update.DirectUpdateHandler2 commit INFO: start
>>>>> > commit(optimize=true,waitFlush=false,waitSearcher=true)
>>>>> >
>>>>> > But I didn't fire any optimize, my delta-import is fired like :
>>>>> > /solr/video/dataimport?command=delta-import&optimize=false
>>>>> >
>>>>> > Solrconfig.xml : autocommit turnd off
>>>>> > <!--<luceneAutoCommit>false</luceneAutoCommit>-->
>>>>> > <!-- <commitLockTimeout>18500</commitLockTimeout>-->
>>>>> > <!-- autocommit pending docs if certain criteria are met
>>>>> > <autoCommit>
>>>>> > <maxDocs>10000</maxDocs>
>>>>> > </autoCommit>
>>>>> > -->
>>>>> >
>>>>> > Maybe it comes from lucene parameters?
>>>>> > <!-- options specific to the main on-disk lucene index -->
>>>>> > <useCompoundFile>false</useCompoundFile>
>>>>> > <ramBufferSizeMB>50</ramBufferSizeMB>
>>>>> > <mergeFactor>50</mergeFactor>
>>>>> > <!-- Deprecated -->
>>>>> > <!--<maxBufferedDocs>1000</maxBufferedDocs>-->
>>>>> > <maxMergeDocs>2147483647</maxMergeDocs>
>>>>> > <maxFieldLength>10000</maxFieldLength>
>>>>> >
>>>>> > Thanks a lot for your help,
>>>>> > Sunny
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22649412.html
>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22661545.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> --Noble Paul
>
>
--
View this message in context: http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22675729.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem for replication : segment optimized automaticly
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
if the DIH status does not say that it optimized, it is lucene
mergeing the segments
On Mon, Mar 23, 2009 at 8:15 PM, sunnyfr <jo...@gmail.com> wrote:
>
> I checked this out but It doesn't say nothing about optimizing.
> I'm sure it's lucene part about merging or I don't know ...??
>
>
> Noble Paul നോബിള് नोब्ळ् wrote:
>>
>> the easiest way to find out what DIH did is to hit it's status
>> command. It will give you a brief description of what all it did
>> during last import
>>
>> On Mon, Mar 23, 2009 at 12:59 AM, Shalin Shekhar Mangar
>> <sh...@gmail.com> wrote:
>>> Lucene will automatically merge segments when they exceed the
>>> mergeFactor.
>>> This may be one reason but I'm not sure.
>>>
>>> I checked DataImportHandler's code again. It won't optimize if
>>> optimize=false is specified.
>>>
>>> On Mon, Mar 23, 2009 at 12:43 AM, sunnyfr <jo...@gmail.com> wrote:
>>>
>>>>
>>>> Do you have any idea ???
>>>> :(
>>>>
>>>> cheer,
>>>>
>>>>
>>>> sunnyfr wrote:
>>>> >
>>>> > Hi everybody ... still me :)
>>>> > hoo happy day :)
>>>> >
>>>> > Just, I dont get where I miss something, I will try to be clear.
>>>> >
>>>> > this is my index folder (and we can notice the evolution according to
>>>> the
>>>> > delta import every 30mn) :
>>>> >
>>>> > root@search-01:/data/solr# ls video/data/index/
>>>> > _2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx
>>>> > _2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx
>>>> > _2ben.frq _2ben.prx _2ben.tis _2beo.fdx segments.gen
>>>> > _2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del
>>>> > _2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm
>>>> > _2ben.nrm _2ben.tii _2beo.fdt _2beo.fnm segments_230x
>>>> > root@search-01:/data/solr# ls video/data/index/
>>>> > _2bel.fdt _2bel.frq _2bel.tii _2bel.tvf _2bem.fdt _2bem.frq
>>>> > _2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm
>>>> > _2beo.tis _2beo.tvx
>>>> > _2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx _2bem.fdx _2bem.nrm
>>>> > _2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx
>>>> > _2beo.tvd segments.gen
>>>> > _2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx
>>>> > _2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii
>>>> > _2beo.tvf segments_230x
>>>> > root@search-01:/data/solr# ls video/data/index/
>>>> > _2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx
>>>> > _2beo.tii _2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen
>>>> > segments_230y
>>>> >
>>>> > So as you can notice my segments increased which is perfect for my
>>>> > replication (faster to get JUST last segments)
>>>> > But like you can see in my last ls my segment has been optimized.
>>>> >
>>>> > Like I can notice in my log :
>>>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
>>>> > org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import
>>>> > completed successfully without optimization
>>>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
>>>> > org.apache.solr.update.DirectUpdateHandler2 commit INFO: start
>>>> > commit(optimize=true,waitFlush=false,waitSearcher=true)
>>>> >
>>>> > But I didn't fire any optimize, my delta-import is fired like :
>>>> > /solr/video/dataimport?command=delta-import&optimize=false
>>>> >
>>>> > Solrconfig.xml : autocommit turnd off
>>>> > <!--<luceneAutoCommit>false</luceneAutoCommit>-->
>>>> > <!-- <commitLockTimeout>18500</commitLockTimeout>-->
>>>> > <!-- autocommit pending docs if certain criteria are met
>>>> > <autoCommit>
>>>> > <maxDocs>10000</maxDocs>
>>>> > </autoCommit>
>>>> > -->
>>>> >
>>>> > Maybe it comes from lucene parameters?
>>>> > <!-- options specific to the main on-disk lucene index -->
>>>> > <useCompoundFile>false</useCompoundFile>
>>>> > <ramBufferSizeMB>50</ramBufferSizeMB>
>>>> > <mergeFactor>50</mergeFactor>
>>>> > <!-- Deprecated -->
>>>> > <!--<maxBufferedDocs>1000</maxBufferedDocs>-->
>>>> > <maxMergeDocs>2147483647</maxMergeDocs>
>>>> > <maxFieldLength>10000</maxFieldLength>
>>>> >
>>>> > Thanks a lot for your help,
>>>> > Sunny
>>>> >
>>>> >
>>>> >
>>>> >
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22649412.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22661545.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: Problem for replication : segment optimized automaticly
Posted by sunnyfr <jo...@gmail.com>.
I checked this out but It doesn't say nothing about optimizing.
I'm sure it's lucene part about merging or I don't know ...??
Noble Paul നോബിള് नोब्ळ् wrote:
>
> the easiest way to find out what DIH did is to hit it's status
> command. It will give you a brief description of what all it did
> during last import
>
> On Mon, Mar 23, 2009 at 12:59 AM, Shalin Shekhar Mangar
> <sh...@gmail.com> wrote:
>> Lucene will automatically merge segments when they exceed the
>> mergeFactor.
>> This may be one reason but I'm not sure.
>>
>> I checked DataImportHandler's code again. It won't optimize if
>> optimize=false is specified.
>>
>> On Mon, Mar 23, 2009 at 12:43 AM, sunnyfr <jo...@gmail.com> wrote:
>>
>>>
>>> Do you have any idea ???
>>> :(
>>>
>>> cheer,
>>>
>>>
>>> sunnyfr wrote:
>>> >
>>> > Hi everybody ... still me :)
>>> > hoo happy day :)
>>> >
>>> > Just, I dont get where I miss something, I will try to be clear.
>>> >
>>> > this is my index folder (and we can notice the evolution according to
>>> the
>>> > delta import every 30mn) :
>>> >
>>> > root@search-01:/data/solr# ls video/data/index/
>>> > _2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx
>>> > _2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx
>>> > _2ben.frq _2ben.prx _2ben.tis _2beo.fdx segments.gen
>>> > _2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del
>>> > _2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm
>>> > _2ben.nrm _2ben.tii _2beo.fdt _2beo.fnm segments_230x
>>> > root@search-01:/data/solr# ls video/data/index/
>>> > _2bel.fdt _2bel.frq _2bel.tii _2bel.tvf _2bem.fdt _2bem.frq
>>> > _2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm
>>> > _2beo.tis _2beo.tvx
>>> > _2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx _2bem.fdx _2bem.nrm
>>> > _2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx
>>> > _2beo.tvd segments.gen
>>> > _2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx
>>> > _2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii
>>> > _2beo.tvf segments_230x
>>> > root@search-01:/data/solr# ls video/data/index/
>>> > _2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx
>>> > _2beo.tii _2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen
>>> > segments_230y
>>> >
>>> > So as you can notice my segments increased which is perfect for my
>>> > replication (faster to get JUST last segments)
>>> > But like you can see in my last ls my segment has been optimized.
>>> >
>>> > Like I can notice in my log :
>>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
>>> > org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import
>>> > completed successfully without optimization
>>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
>>> > org.apache.solr.update.DirectUpdateHandler2 commit INFO: start
>>> > commit(optimize=true,waitFlush=false,waitSearcher=true)
>>> >
>>> > But I didn't fire any optimize, my delta-import is fired like :
>>> > /solr/video/dataimport?command=delta-import&optimize=false
>>> >
>>> > Solrconfig.xml : autocommit turnd off
>>> > <!--<luceneAutoCommit>false</luceneAutoCommit>-->
>>> > <!-- <commitLockTimeout>18500</commitLockTimeout>-->
>>> > <!-- autocommit pending docs if certain criteria are met
>>> > <autoCommit>
>>> > <maxDocs>10000</maxDocs>
>>> > </autoCommit>
>>> > -->
>>> >
>>> > Maybe it comes from lucene parameters?
>>> > <!-- options specific to the main on-disk lucene index -->
>>> > <useCompoundFile>false</useCompoundFile>
>>> > <ramBufferSizeMB>50</ramBufferSizeMB>
>>> > <mergeFactor>50</mergeFactor>
>>> > <!-- Deprecated -->
>>> > <!--<maxBufferedDocs>1000</maxBufferedDocs>-->
>>> > <maxMergeDocs>2147483647</maxMergeDocs>
>>> > <maxFieldLength>10000</maxFieldLength>
>>> >
>>> > Thanks a lot for your help,
>>> > Sunny
>>> >
>>> >
>>> >
>>> >
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22649412.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
>
>
> --
> --Noble Paul
>
>
--
View this message in context: http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22661545.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem for replication : segment optimized automaticly
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
the easiest way to find out what DIH did is to hit it's status
command. It will give you a brief description of what all it did
during last import
On Mon, Mar 23, 2009 at 12:59 AM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
> Lucene will automatically merge segments when they exceed the mergeFactor.
> This may be one reason but I'm not sure.
>
> I checked DataImportHandler's code again. It won't optimize if
> optimize=false is specified.
>
> On Mon, Mar 23, 2009 at 12:43 AM, sunnyfr <jo...@gmail.com> wrote:
>
>>
>> Do you have any idea ???
>> :(
>>
>> cheer,
>>
>>
>> sunnyfr wrote:
>> >
>> > Hi everybody ... still me :)
>> > hoo happy day :)
>> >
>> > Just, I dont get where I miss something, I will try to be clear.
>> >
>> > this is my index folder (and we can notice the evolution according to the
>> > delta import every 30mn) :
>> >
>> > root@search-01:/data/solr# ls video/data/index/
>> > _2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx
>> > _2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx
>> > _2ben.frq _2ben.prx _2ben.tis _2beo.fdx segments.gen
>> > _2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del
>> > _2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm
>> > _2ben.nrm _2ben.tii _2beo.fdt _2beo.fnm segments_230x
>> > root@search-01:/data/solr# ls video/data/index/
>> > _2bel.fdt _2bel.frq _2bel.tii _2bel.tvf _2bem.fdt _2bem.frq
>> > _2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm
>> > _2beo.tis _2beo.tvx
>> > _2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx _2bem.fdx _2bem.nrm
>> > _2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx
>> > _2beo.tvd segments.gen
>> > _2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx
>> > _2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii
>> > _2beo.tvf segments_230x
>> > root@search-01:/data/solr# ls video/data/index/
>> > _2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx
>> > _2beo.tii _2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen
>> > segments_230y
>> >
>> > So as you can notice my segments increased which is perfect for my
>> > replication (faster to get JUST last segments)
>> > But like you can see in my last ls my segment has been optimized.
>> >
>> > Like I can notice in my log :
>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
>> > org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import
>> > completed successfully without optimization
>> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
>> > org.apache.solr.update.DirectUpdateHandler2 commit INFO: start
>> > commit(optimize=true,waitFlush=false,waitSearcher=true)
>> >
>> > But I didn't fire any optimize, my delta-import is fired like :
>> > /solr/video/dataimport?command=delta-import&optimize=false
>> >
>> > Solrconfig.xml : autocommit turnd off
>> > <!--<luceneAutoCommit>false</luceneAutoCommit>-->
>> > <!-- <commitLockTimeout>18500</commitLockTimeout>-->
>> > <!-- autocommit pending docs if certain criteria are met
>> > <autoCommit>
>> > <maxDocs>10000</maxDocs>
>> > </autoCommit>
>> > -->
>> >
>> > Maybe it comes from lucene parameters?
>> > <!-- options specific to the main on-disk lucene index -->
>> > <useCompoundFile>false</useCompoundFile>
>> > <ramBufferSizeMB>50</ramBufferSizeMB>
>> > <mergeFactor>50</mergeFactor>
>> > <!-- Deprecated -->
>> > <!--<maxBufferedDocs>1000</maxBufferedDocs>-->
>> > <maxMergeDocs>2147483647</maxMergeDocs>
>> > <maxFieldLength>10000</maxFieldLength>
>> >
>> > Thanks a lot for your help,
>> > Sunny
>> >
>> >
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22649412.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
--
--Noble Paul
Re: Problem for replication : segment optimized automaticly
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Lucene will automatically merge segments when they exceed the mergeFactor.
This may be one reason but I'm not sure.
I checked DataImportHandler's code again. It won't optimize if
optimize=false is specified.
On Mon, Mar 23, 2009 at 12:43 AM, sunnyfr <jo...@gmail.com> wrote:
>
> Do you have any idea ???
> :(
>
> cheer,
>
>
> sunnyfr wrote:
> >
> > Hi everybody ... still me :)
> > hoo happy day :)
> >
> > Just, I dont get where I miss something, I will try to be clear.
> >
> > this is my index folder (and we can notice the evolution according to the
> > delta import every 30mn) :
> >
> > root@search-01:/data/solr# ls video/data/index/
> > _2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx
> > _2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx
> > _2ben.frq _2ben.prx _2ben.tis _2beo.fdx segments.gen
> > _2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del
> > _2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm
> > _2ben.nrm _2ben.tii _2beo.fdt _2beo.fnm segments_230x
> > root@search-01:/data/solr# ls video/data/index/
> > _2bel.fdt _2bel.frq _2bel.tii _2bel.tvf _2bem.fdt _2bem.frq
> > _2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm
> > _2beo.tis _2beo.tvx
> > _2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx _2bem.fdx _2bem.nrm
> > _2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx
> > _2beo.tvd segments.gen
> > _2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx
> > _2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii
> > _2beo.tvf segments_230x
> > root@search-01:/data/solr# ls video/data/index/
> > _2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx
> > _2beo.tii _2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen
> > segments_230y
> >
> > So as you can notice my segments increased which is perfect for my
> > replication (faster to get JUST last segments)
> > But like you can see in my last ls my segment has been optimized.
> >
> > Like I can notice in my log :
> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
> > org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import
> > completed successfully without optimization
> > Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
> > org.apache.solr.update.DirectUpdateHandler2 commit INFO: start
> > commit(optimize=true,waitFlush=false,waitSearcher=true)
> >
> > But I didn't fire any optimize, my delta-import is fired like :
> > /solr/video/dataimport?command=delta-import&optimize=false
> >
> > Solrconfig.xml : autocommit turnd off
> > <!--<luceneAutoCommit>false</luceneAutoCommit>-->
> > <!-- <commitLockTimeout>18500</commitLockTimeout>-->
> > <!-- autocommit pending docs if certain criteria are met
> > <autoCommit>
> > <maxDocs>10000</maxDocs>
> > </autoCommit>
> > -->
> >
> > Maybe it comes from lucene parameters?
> > <!-- options specific to the main on-disk lucene index -->
> > <useCompoundFile>false</useCompoundFile>
> > <ramBufferSizeMB>50</ramBufferSizeMB>
> > <mergeFactor>50</mergeFactor>
> > <!-- Deprecated -->
> > <!--<maxBufferedDocs>1000</maxBufferedDocs>-->
> > <maxMergeDocs>2147483647</maxMergeDocs>
> > <maxFieldLength>10000</maxFieldLength>
> >
> > Thanks a lot for your help,
> > Sunny
> >
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22649412.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: Problem for replication : segment optimized automaticly
Posted by sunnyfr <jo...@gmail.com>.
Do you have any idea ???
:(
cheer,
sunnyfr wrote:
>
> Hi everybody ... still me :)
> hoo happy day :)
>
> Just, I dont get where I miss something, I will try to be clear.
>
> this is my index folder (and we can notice the evolution according to the
> delta import every 30mn) :
>
> root@search-01:/data/solr# ls video/data/index/
> _2bel.fdt _2bel.fnm _2bel.nrm _2bel.tii _2bel.tvd _2bel.tvx
> _2bem.fdt _2bem.fnm _2bem.nrm _2bem.tii _2bem.tvd _2bem.tvx
> _2ben.frq _2ben.prx _2ben.tis _2beo.fdx segments.gen
> _2bel.fdx _2bel.frq _2bel.prx _2bel.tis _2bel.tvf _2bel_1.del
> _2bem.fdx _2bem.frq _2bem.prx _2bem.tis _2bem.tvf _2ben.fnm
> _2ben.nrm _2ben.tii _2beo.fdt _2beo.fnm segments_230x
> root@search-01:/data/solr# ls video/data/index/
> _2bel.fdt _2bel.frq _2bel.tii _2bel.tvf _2bem.fdt _2bem.frq
> _2bem.tii _2bem.tvf _2ben.frq _2ben.tii _2beo.fdx _2beo.nrm
> _2beo.tis _2beo.tvx
> _2bel.fdx _2bel.nrm _2bel.tis _2bel.tvx _2bem.fdx _2bem.nrm
> _2bem.tis _2bem.tvx _2ben.nrm _2ben.tis _2beo.fnm _2beo.prx
> _2beo.tvd segments.gen
> _2bel.fnm _2bel.prx _2bel.tvd _2bel_1.del _2bem.fnm _2bem.prx
> _2bem.tvd _2ben.fnm _2ben.prx _2beo.fdt _2beo.frq _2beo.tii
> _2beo.tvf segments_230x
> root@search-01:/data/solr# ls video/data/index/
> _2beo.fdt _2beo.fdx _2beo.fnm _2beo.frq _2beo.nrm _2beo.prx
> _2beo.tii _2beo.tis _2beo.tvd _2beo.tvf _2beo.tvx segments.gen
> segments_230y
>
> So as you can notice my segments increased which is perfect for my
> replication (faster to get JUST last segments)
> But like you can see in my last ls my segment has been optimized.
>
> Like I can notice in my log :
> Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
> org.apache.solr.handler.dataimport.DocBuilder commit INFO: Full Import
> completed successfully without optimization
> Mar 19 15:42:37 search-01 jsvc.exec[23255]: Mar 19, 2009 3:42:37 PM
> org.apache.solr.update.DirectUpdateHandler2 commit INFO: start
> commit(optimize=true,waitFlush=false,waitSearcher=true)
>
> But I didn't fire any optimize, my delta-import is fired like :
> /solr/video/dataimport?command=delta-import&optimize=false
>
> Solrconfig.xml : autocommit turnd off
> <!--<luceneAutoCommit>false</luceneAutoCommit>-->
> <!-- <commitLockTimeout>18500</commitLockTimeout>-->
> <!-- autocommit pending docs if certain criteria are met
> <autoCommit>
> <maxDocs>10000</maxDocs>
> </autoCommit>
> -->
>
> Maybe it comes from lucene parameters?
> <!-- options specific to the main on-disk lucene index -->
> <useCompoundFile>false</useCompoundFile>
> <ramBufferSizeMB>50</ramBufferSizeMB>
> <mergeFactor>50</mergeFactor>
> <!-- Deprecated -->
> <!--<maxBufferedDocs>1000</maxBufferedDocs>-->
> <maxMergeDocs>2147483647</maxMergeDocs>
> <maxFieldLength>10000</maxFieldLength>
>
> Thanks a lot for your help,
> Sunny
>
>
>
>
--
View this message in context: http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-tp22601442p22649412.html
Sent from the Solr - User mailing list archive at Nabble.com.