You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Jason, Kim" <hi...@gmail.com> on 2011/06/20 02:50:32 UTC
why too many open files?
Hi, All
I have 12 shards and ramBufferSizeMB=512, mergeFactor=5.
But solr raise java.io.FileNotFoundException (Too many open files).
mergeFactor is just 5. How can this happen?
Below is segments of some shard. That is too many segments over mergFactor.
What's wrong and How should I set the mergeFactor?
==========================================================
[root@solr solr]# ls indexData/multicore-us/usn02/data/index/
_0.fdt _gs.fdt _h5.tii _hl.nrm _i1.nrm _kn.nrm _l1.nrm _lq.tii
_0.fdx _gs.fdx _h5.tis _hl.prx _i1.prx _kn.prx _l1.prx _lq.tis
_3i.fdt _gs.fnm _h7.fnm _hl.tii _i1.tii _kn.tii _l1.tii
lucene-2de7b31b5eabdff0b6ec7fd32eecf8c7-write.lock
_3i.fdx _gs.frq _h7.frq _hl.tis _i1.tis _kn.tis _l1.tis _lu.fnm
_3s.fnm _gs.nrm _h7.nrm _hn.fnm _j7.fdt _kp.fnm _l2.fnm _lu.frq
_3s.frq _gs.prx _h7.prx _hn.frq _j7.fdx _kp.frq _l2.frq _lu.nrm
_3s.nrm _gs.tii _h7.tii _hn.nrm _kb.fnm _kp.nrm _l2.nrm _lu.prx
_3s.prx _gs.tis _h7.tis _hn.prx _kb.frq _kp.prx _l2.prx _lu.tii
_3s.tii _gu.fnm _h9.fnm _hn.tii _kb.nrm _kp.tii _l2.tii _lu.tis
_3s.tis _gu.frq _h9.frq _hn.tis _kb.prx _kp.tis _l2.tis _ly.fnm
_48.fdt _gu.nrm _h9.nrm _hp.fnm _kb.tii _kq.fnm _l6.fnm _ly.frq
_48.fdx _gu.prx _h9.prx _hp.frq _kb.tis _kq.frq _l6.frq _ly.nrm
_4d.fnm _gu.tii _h9.tii _hp.nrm _kc.fnm _kq.nrm _l6.nrm _ly.prx
_4d.frq _gu.tis _h9.tis _hp.prx _kc.frq _kq.prx _l6.prx _ly.tii
_4d.nrm _gw.fnm _hb.fnm _hp.tii _kc.nrm _kq.tii _l6.tii _ly.tis
_4d.prx _gw.frq _hb.frq _hp.tis _kc.prx _kq.tis _l6.tis _m3.fnm
_4d.tii _gw.nrm _hb.nrm _hr.fnm _kc.tii _kr.fnm _la.fnm _m3.frq
_4d.tis _gw.prx _hb.prx _hr.frq _kc.tis _kr.frq _la.frq _m3.nrm
_5b.fdt _gw.tii _hb.tii _hr.nrm _kf.fdt _kr.nrm _la.nrm _m3.prx
_5b.fdx _gw.tis _hb.tis _hr.prx _kf.fdx _kr.prx _la.prx _m3.tii
_5b.fnm _gy.fnm _he.fdt _hr.tii _kf.fnm _kr.tii _la.tii _m3.tis
_5b.frq _gy.frq _he.fdx _hr.tis _kf.frq _kr.tis _la.tis _m8.fnm
_5b.nrm _gy.nrm _he.fnm _ht.fnm _kf.nrm _kt.fnm _le.fnm _m8.frq
_5b.prx _gy.prx _he.frq _ht.frq _kf.prx _kt.frq _le.frq _m8.nrm
_5b.tii _gy.tii _he.nrm _ht.nrm _kf.tii _kt.nrm _le.nrm _m8.prx
_5b.tis _gy.tis _he.prx _ht.prx _kf.tis _kt.prx _le.prx _m8.tii
_5m.fnm _h0.fnm _he.tii _ht.tii _kg.fnm _kt.tii _le.tii _m8.tis
_5m.frq _h0.frq _he.tis _ht.tis _kg.frq _kt.tis _le.tis _md.fnm
_5m.nrm _h0.nrm _hh.fnm _hv.fnm _kg.nrm _kw.fnm _li.fnm _md.frq
_5m.prx _h0.prx _hh.frq _hv.frq _kg.prx _kw.frq _li.frq _md.nrm
_5m.tii _h0.tii _hh.nrm _hv.nrm _kg.tii _kw.nrm _li.nrm _md.prx
_5m.tis _h0.tis _hh.prx _hv.prx _kg.tis _kw.prx _li.prx _md.tii
_5n.fnm _h2.fnm _hh.tii _hv.tii _kj.fdt _kw.tii _li.tii _md.tis
_5n.frq _h2.frq _hh.tis _hv.tis _kj.fdx _kw.tis _li.tis _mi.fnm
_5n.nrm _h2.nrm _hk.fnm _hz.fdt _kj.fnm _ky.fnm _lm.fnm _mi.frq
_5n.prx _h2.prx _hk.frq _hz.fdx _kj.frq _ky.frq _lm.frq _mi.nrm
_5n.tii _h2.tii _hk.nrm _hz.fnm _kj.nrm _ky.nrm _lm.nrm _mi.prx
_5n.tis _h2.tis _hk.prx _hz.frq _kj.prx _ky.prx _lm.prx _mi.tii
_5x.fnm _h5.fdt _hk.tii _hz.nrm _kj.tii _ky.tii _lm.tii _mi.tis
_5x.frq _h5.fdx _hk.tis _hz.prx _kj.tis _ky.tis _lm.tis segments_1
_5x.nrm _h5.fnm _hl.fdt _hz.tii _kn.fdt _l1.fdt _lq.fnm segments.gen
_5x.prx _h5.frq _hl.fdx _hz.tis _kn.fdx _l1.fdx _lq.frq
_5x.tii _h5.nrm _hl.fnm _i1.fnm _kn.fnm _l1.fnm _lq.nrm
_5x.tis _h5.prx _hl.frq _i1.frq _kn.frq _l1.frq _lq.prx
==========================================================
Thanks in advance.
--
View this message in context: http://lucene.472066.n3.nabble.com/why-too-many-open-files-tp3084407p3084407.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: why too many open files?
Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
(11/06/20 16:16), Jason, Kim wrote:
> Hi, Mark
>
> I think FileNotFoundException will be worked around by raise the ulimit.
> I just want to know why segments are created more than mergeFactor.
> During the googling, I found contents concerning mergeFactor:
> http://web.archiveorange.com/archive/v/bH0vUQzfYcdtZoocG2C9
> Yonik wrote:
> "mergeFactor 10 means a maximum of 10 segments at each "level".
> if maxBufferedDocs=10 with a log doc merge policy (equivalent to
> Lucene in the old days), then you could have up to ~ 10*log10(nDocs)
> segments in the index (i.e. up to 60 segments for a 1M doc index)."
>
> But I don't understand this.
> someone explain to me in more detail?
Take a look at:
Visualizing Lucene's segment merges
http://s.apache.org/merging
koji
--
http://www.rondhuit.com/en/
Re: why too many open files?
Posted by "Jason, Kim" <hi...@gmail.com>.
Hi, Mark
I think FileNotFoundException will be worked around by raise the ulimit.
I just want to know why segments are created more than mergeFactor.
During the googling, I found contents concerning mergeFactor:
http://web.archiveorange.com/archive/v/bH0vUQzfYcdtZoocG2C9
Yonik wrote:
"mergeFactor 10 means a maximum of 10 segments at each "level".
if maxBufferedDocs=10 with a log doc merge policy (equivalent to
Lucene in the old days), then you could have up to ~ 10*log10(nDocs)
segments in the index (i.e. up to 60 segments for a 1M doc index)."
But I don't understand this.
someone explain to me in more detail?
Thanks
--
View this message in context: http://lucene.472066.n3.nabble.com/why-too-many-open-files-tp3084407p3085172.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: why too many open files?
Posted by Mark Schoy <he...@gmx.de>.
Hi,
did you have checked the max opened files of your OS?
see: http://lj4newbies.blogspot.com/2007/04/too-many-open-files.html
2011/6/20 Jason, Kim <hi...@gmail.com>
> Hi, All
>
> I have 12 shards and ramBufferSizeMB=512, mergeFactor=5.
> But solr raise java.io.FileNotFoundException (Too many open files).
> mergeFactor is just 5. How can this happen?
> Below is segments of some shard. That is too many segments over mergFactor.
> What's wrong and How should I set the mergeFactor?
>
> ==========================================================
> [root@solr solr]# ls indexData/multicore-us/usn02/data/index/
> _0.fdt _gs.fdt _h5.tii _hl.nrm _i1.nrm _kn.nrm _l1.nrm _lq.tii
> _0.fdx _gs.fdx _h5.tis _hl.prx _i1.prx _kn.prx _l1.prx _lq.tis
> _3i.fdt _gs.fnm _h7.fnm _hl.tii _i1.tii _kn.tii _l1.tii
> lucene-2de7b31b5eabdff0b6ec7fd32eecf8c7-write.lock
> _3i.fdx _gs.frq _h7.frq _hl.tis _i1.tis _kn.tis _l1.tis _lu.fnm
> _3s.fnm _gs.nrm _h7.nrm _hn.fnm _j7.fdt _kp.fnm _l2.fnm _lu.frq
> _3s.frq _gs.prx _h7.prx _hn.frq _j7.fdx _kp.frq _l2.frq _lu.nrm
> _3s.nrm _gs.tii _h7.tii _hn.nrm _kb.fnm _kp.nrm _l2.nrm _lu.prx
> _3s.prx _gs.tis _h7.tis _hn.prx _kb.frq _kp.prx _l2.prx _lu.tii
> _3s.tii _gu.fnm _h9.fnm _hn.tii _kb.nrm _kp.tii _l2.tii _lu.tis
> _3s.tis _gu.frq _h9.frq _hn.tis _kb.prx _kp.tis _l2.tis _ly.fnm
> _48.fdt _gu.nrm _h9.nrm _hp.fnm _kb.tii _kq.fnm _l6.fnm _ly.frq
> _48.fdx _gu.prx _h9.prx _hp.frq _kb.tis _kq.frq _l6.frq _ly.nrm
> _4d.fnm _gu.tii _h9.tii _hp.nrm _kc.fnm _kq.nrm _l6.nrm _ly.prx
> _4d.frq _gu.tis _h9.tis _hp.prx _kc.frq _kq.prx _l6.prx _ly.tii
> _4d.nrm _gw.fnm _hb.fnm _hp.tii _kc.nrm _kq.tii _l6.tii _ly.tis
> _4d.prx _gw.frq _hb.frq _hp.tis _kc.prx _kq.tis _l6.tis _m3.fnm
> _4d.tii _gw.nrm _hb.nrm _hr.fnm _kc.tii _kr.fnm _la.fnm _m3.frq
> _4d.tis _gw.prx _hb.prx _hr.frq _kc.tis _kr.frq _la.frq _m3.nrm
> _5b.fdt _gw.tii _hb.tii _hr.nrm _kf.fdt _kr.nrm _la.nrm _m3.prx
> _5b.fdx _gw.tis _hb.tis _hr.prx _kf.fdx _kr.prx _la.prx _m3.tii
> _5b.fnm _gy.fnm _he.fdt _hr.tii _kf.fnm _kr.tii _la.tii _m3.tis
> _5b.frq _gy.frq _he.fdx _hr.tis _kf.frq _kr.tis _la.tis _m8.fnm
> _5b.nrm _gy.nrm _he.fnm _ht.fnm _kf.nrm _kt.fnm _le.fnm _m8.frq
> _5b.prx _gy.prx _he.frq _ht.frq _kf.prx _kt.frq _le.frq _m8.nrm
> _5b.tii _gy.tii _he.nrm _ht.nrm _kf.tii _kt.nrm _le.nrm _m8.prx
> _5b.tis _gy.tis _he.prx _ht.prx _kf.tis _kt.prx _le.prx _m8.tii
> _5m.fnm _h0.fnm _he.tii _ht.tii _kg.fnm _kt.tii _le.tii _m8.tis
> _5m.frq _h0.frq _he.tis _ht.tis _kg.frq _kt.tis _le.tis _md.fnm
> _5m.nrm _h0.nrm _hh.fnm _hv.fnm _kg.nrm _kw.fnm _li.fnm _md.frq
> _5m.prx _h0.prx _hh.frq _hv.frq _kg.prx _kw.frq _li.frq _md.nrm
> _5m.tii _h0.tii _hh.nrm _hv.nrm _kg.tii _kw.nrm _li.nrm _md.prx
> _5m.tis _h0.tis _hh.prx _hv.prx _kg.tis _kw.prx _li.prx _md.tii
> _5n.fnm _h2.fnm _hh.tii _hv.tii _kj.fdt _kw.tii _li.tii _md.tis
> _5n.frq _h2.frq _hh.tis _hv.tis _kj.fdx _kw.tis _li.tis _mi.fnm
> _5n.nrm _h2.nrm _hk.fnm _hz.fdt _kj.fnm _ky.fnm _lm.fnm _mi.frq
> _5n.prx _h2.prx _hk.frq _hz.fdx _kj.frq _ky.frq _lm.frq _mi.nrm
> _5n.tii _h2.tii _hk.nrm _hz.fnm _kj.nrm _ky.nrm _lm.nrm _mi.prx
> _5n.tis _h2.tis _hk.prx _hz.frq _kj.prx _ky.prx _lm.prx _mi.tii
> _5x.fnm _h5.fdt _hk.tii _hz.nrm _kj.tii _ky.tii _lm.tii _mi.tis
> _5x.frq _h5.fdx _hk.tis _hz.prx _kj.tis _ky.tis _lm.tis segments_1
> _5x.nrm _h5.fnm _hl.fdt _hz.tii _kn.fdt _l1.fdt _lq.fnm segments.gen
> _5x.prx _h5.frq _hl.fdx _hz.tis _kn.fdx _l1.fdx _lq.frq
> _5x.tii _h5.nrm _hl.fnm _i1.fnm _kn.fnm _l1.fnm _lq.nrm
> _5x.tis _h5.prx _hl.frq _i1.frq _kn.frq _l1.frq _lq.prx
> ==========================================================
>
> Thanks in advance.
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/why-too-many-open-files-tp3084407p3084407.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: why too many open files?
Posted by Markus Jelsma <ma...@openindex.io>.
12 shards on the same machine?
> Hi, All
>
> I have 12 shards and ramBufferSizeMB=512, mergeFactor=5.
> But solr raise java.io.FileNotFoundException (Too many open files).
> mergeFactor is just 5. How can this happen?
> Below is segments of some shard. That is too many segments over mergFactor.
> What's wrong and How should I set the mergeFactor?
>
> ==========================================================
> [root@solr solr]# ls indexData/multicore-us/usn02/data/index/
> _0.fdt _gs.fdt _h5.tii _hl.nrm _i1.nrm _kn.nrm _l1.nrm _lq.tii
> _0.fdx _gs.fdx _h5.tis _hl.prx _i1.prx _kn.prx _l1.prx _lq.tis
> _3i.fdt _gs.fnm _h7.fnm _hl.tii _i1.tii _kn.tii _l1.tii
> lucene-2de7b31b5eabdff0b6ec7fd32eecf8c7-write.lock
> _3i.fdx _gs.frq _h7.frq _hl.tis _i1.tis _kn.tis _l1.tis _lu.fnm
> _3s.fnm _gs.nrm _h7.nrm _hn.fnm _j7.fdt _kp.fnm _l2.fnm _lu.frq
> _3s.frq _gs.prx _h7.prx _hn.frq _j7.fdx _kp.frq _l2.frq _lu.nrm
> _3s.nrm _gs.tii _h7.tii _hn.nrm _kb.fnm _kp.nrm _l2.nrm _lu.prx
> _3s.prx _gs.tis _h7.tis _hn.prx _kb.frq _kp.prx _l2.prx _lu.tii
> _3s.tii _gu.fnm _h9.fnm _hn.tii _kb.nrm _kp.tii _l2.tii _lu.tis
> _3s.tis _gu.frq _h9.frq _hn.tis _kb.prx _kp.tis _l2.tis _ly.fnm
> _48.fdt _gu.nrm _h9.nrm _hp.fnm _kb.tii _kq.fnm _l6.fnm _ly.frq
> _48.fdx _gu.prx _h9.prx _hp.frq _kb.tis _kq.frq _l6.frq _ly.nrm
> _4d.fnm _gu.tii _h9.tii _hp.nrm _kc.fnm _kq.nrm _l6.nrm _ly.prx
> _4d.frq _gu.tis _h9.tis _hp.prx _kc.frq _kq.prx _l6.prx _ly.tii
> _4d.nrm _gw.fnm _hb.fnm _hp.tii _kc.nrm _kq.tii _l6.tii _ly.tis
> _4d.prx _gw.frq _hb.frq _hp.tis _kc.prx _kq.tis _l6.tis _m3.fnm
> _4d.tii _gw.nrm _hb.nrm _hr.fnm _kc.tii _kr.fnm _la.fnm _m3.frq
> _4d.tis _gw.prx _hb.prx _hr.frq _kc.tis _kr.frq _la.frq _m3.nrm
> _5b.fdt _gw.tii _hb.tii _hr.nrm _kf.fdt _kr.nrm _la.nrm _m3.prx
> _5b.fdx _gw.tis _hb.tis _hr.prx _kf.fdx _kr.prx _la.prx _m3.tii
> _5b.fnm _gy.fnm _he.fdt _hr.tii _kf.fnm _kr.tii _la.tii _m3.tis
> _5b.frq _gy.frq _he.fdx _hr.tis _kf.frq _kr.tis _la.tis _m8.fnm
> _5b.nrm _gy.nrm _he.fnm _ht.fnm _kf.nrm _kt.fnm _le.fnm _m8.frq
> _5b.prx _gy.prx _he.frq _ht.frq _kf.prx _kt.frq _le.frq _m8.nrm
> _5b.tii _gy.tii _he.nrm _ht.nrm _kf.tii _kt.nrm _le.nrm _m8.prx
> _5b.tis _gy.tis _he.prx _ht.prx _kf.tis _kt.prx _le.prx _m8.tii
> _5m.fnm _h0.fnm _he.tii _ht.tii _kg.fnm _kt.tii _le.tii _m8.tis
> _5m.frq _h0.frq _he.tis _ht.tis _kg.frq _kt.tis _le.tis _md.fnm
> _5m.nrm _h0.nrm _hh.fnm _hv.fnm _kg.nrm _kw.fnm _li.fnm _md.frq
> _5m.prx _h0.prx _hh.frq _hv.frq _kg.prx _kw.frq _li.frq _md.nrm
> _5m.tii _h0.tii _hh.nrm _hv.nrm _kg.tii _kw.nrm _li.nrm _md.prx
> _5m.tis _h0.tis _hh.prx _hv.prx _kg.tis _kw.prx _li.prx _md.tii
> _5n.fnm _h2.fnm _hh.tii _hv.tii _kj.fdt _kw.tii _li.tii _md.tis
> _5n.frq _h2.frq _hh.tis _hv.tis _kj.fdx _kw.tis _li.tis _mi.fnm
> _5n.nrm _h2.nrm _hk.fnm _hz.fdt _kj.fnm _ky.fnm _lm.fnm _mi.frq
> _5n.prx _h2.prx _hk.frq _hz.fdx _kj.frq _ky.frq _lm.frq _mi.nrm
> _5n.tii _h2.tii _hk.nrm _hz.fnm _kj.nrm _ky.nrm _lm.nrm _mi.prx
> _5n.tis _h2.tis _hk.prx _hz.frq _kj.prx _ky.prx _lm.prx _mi.tii
> _5x.fnm _h5.fdt _hk.tii _hz.nrm _kj.tii _ky.tii _lm.tii _mi.tis
> _5x.frq _h5.fdx _hk.tis _hz.prx _kj.tis _ky.tis _lm.tis segments_1
> _5x.nrm _h5.fnm _hl.fdt _hz.tii _kn.fdt _l1.fdt _lq.fnm segments.gen
> _5x.prx _h5.frq _hl.fdx _hz.tis _kn.fdx _l1.fdx _lq.frq
> _5x.tii _h5.nrm _hl.fnm _i1.fnm _kn.fnm _l1.fnm _lq.nrm
> _5x.tis _h5.prx _hl.frq _i1.frq _kn.frq _l1.frq _lq.prx
> ==========================================================
>
> Thanks in advance.
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/why-too-many-open-files-tp3084407p30844
> 07.html Sent from the Solr - User mailing list archive at Nabble.com.