You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Kevin Burton <bu...@spinn3r.com> on 2016/11/16 01:32:07 UTC
Possible to cause documents to be contiguous after forceMerge?
I have a large index (say 500GB) that with a large percentage of near
duplicate documents.
I have to keep the documents there (can't delete them) as the metadata is
important.
Is it possible to get the documents to be contiguous somehow?
Once they are contiguous then they will compress very well - which I've
already confirmed by writing the exact same document N times.
IDEALLY I could use two fields and have a unique document ID but then a
group_id so that they can be located on disk by the group_id... but I don't
think this is possible.
Can I just create a synthetic "id" field for this and assume that "id" is
ordered on disk in the lucene index?
--
We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Possible to cause documents to be contiguous after forceMerge?
Posted by Tommaso Teofili <to...@gmail.com>.
improved locality of "near" documents could be used to avoid loading some
segments during the retrieval phase for certain use cases (e.g. spatial
search).
Il giorno mer 16 nov 2016 alle ore 09:45 Ishan Chattopadhyaya <
ichattopadhyaya@gmail.com> ha scritto:
http://shaierera.blogspot.com/2013/04/index-sorting-with-lucene.html
On Wed, Nov 16, 2016 at 11:15 AM, Ishan Chattopadhyaya <
ichattopadhyaya@gmail.com> wrote:
> Can IndexSort help here?
> ------------------------------
> From: Erick Erickson <er...@gmail.com>
> Sent: 11/16/2016 9:29
> To: java-user <ja...@lucene.apache.org>
> Subject: Re: Possible to cause documents to be contiguous after
> forceMerge?
>
> Well, codecs are pluggable so if you can show that you'd get
> an improvement (however you measure them) and that whatever
> you have in mind wouldn't penalize the general case you could
> submit it as a proposal/patch.
>
> Best,
> Erick
>
> On Tue, Nov 15, 2016 at 6:21 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> > On Tue, Nov 15, 2016 at 6:16 PM, Erick Erickson <erickerickson@gmail.com
> >
> > wrote:
> >
> >> You can make no assumptions about locality in terms of where separate
> >> documents land on disk. I suppose if you have the whole corpus at index
> >> time you
> >> could index these "similar" documents contiguously. T
> >>
> >
> > Wow.. that's shockingly frightening. There are a ton of optimizations if
> > you can trick the underlying content store into performing locality.
> >
> > Not trying to be overly negative so another way to phrase it is that at
> > least there's room for improvement !
> >
> >
> >> My base question is why you'd care about compressing 500G. Disk space
> >> is so cheap that the expense of trying to control this dwarfs any
> >> imaginable
> >> $avings, unless you're talking about a lot of 500G indexes. In other
> words
> >> this seems like an
> >> XY problem, you're asking about compressing when you are really
> concerned
> >> with something else.
> >>
> >
> > 500GB per day... additionally, disk is cheap, but IOPS are not. The more
> we
> > can keep in ram and on SSD the better.
> >
> > And we're trying to get as much in RAM then SSD as possible... plus we
> have
> > about 2 years of content. It adds up ;)
> >
> > Kevin
> >
> > --
> >
> > We’re hiring if you know of any awesome Java Devops or Linux Operations
> > Engineers!
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Possible to cause documents to be contiguous after forceMerge?
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
http://shaierera.blogspot.com/2013/04/index-sorting-with-lucene.html
On Wed, Nov 16, 2016 at 11:15 AM, Ishan Chattopadhyaya <
ichattopadhyaya@gmail.com> wrote:
> Can IndexSort help here?
> ------------------------------
> From: Erick Erickson <er...@gmail.com>
> Sent: 11/16/2016 9:29
> To: java-user <ja...@lucene.apache.org>
> Subject: Re: Possible to cause documents to be contiguous after
> forceMerge?
>
> Well, codecs are pluggable so if you can show that you'd get
> an improvement (however you measure them) and that whatever
> you have in mind wouldn't penalize the general case you could
> submit it as a proposal/patch.
>
> Best,
> Erick
>
> On Tue, Nov 15, 2016 at 6:21 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> > On Tue, Nov 15, 2016 at 6:16 PM, Erick Erickson <erickerickson@gmail.com
> >
> > wrote:
> >
> >> You can make no assumptions about locality in terms of where separate
> >> documents land on disk. I suppose if you have the whole corpus at index
> >> time you
> >> could index these "similar" documents contiguously. T
> >>
> >
> > Wow.. that's shockingly frightening. There are a ton of optimizations if
> > you can trick the underlying content store into performing locality.
> >
> > Not trying to be overly negative so another way to phrase it is that at
> > least there's room for improvement !
> >
> >
> >> My base question is why you'd care about compressing 500G. Disk space
> >> is so cheap that the expense of trying to control this dwarfs any
> >> imaginable
> >> $avings, unless you're talking about a lot of 500G indexes. In other
> words
> >> this seems like an
> >> XY problem, you're asking about compressing when you are really
> concerned
> >> with something else.
> >>
> >
> > 500GB per day... additionally, disk is cheap, but IOPS are not. The more
> we
> > can keep in ram and on SSD the better.
> >
> > And we're trying to get as much in RAM then SSD as possible... plus we
> have
> > about 2 years of content. It adds up ;)
> >
> > Kevin
> >
> > --
> >
> > We’re hiring if you know of any awesome Java Devops or Linux Operations
> > Engineers!
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Possible to cause documents to be contiguous after forceMerge?
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
Can IndexSort help here?
-----Original Message-----
From: "Erick Erickson" <er...@gmail.com>
Sent: 11/16/2016 9:29
To: "java-user" <ja...@lucene.apache.org>
Subject: Re: Possible to cause documents to be contiguous after forceMerge?
Well, codecs are pluggable so if you can show that you'd get
an improvement (however you measure them) and that whatever
you have in mind wouldn't penalize the general case you could
submit it as a proposal/patch.
Best,
Erick
On Tue, Nov 15, 2016 at 6:21 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> On Tue, Nov 15, 2016 at 6:16 PM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> You can make no assumptions about locality in terms of where separate
>> documents land on disk. I suppose if you have the whole corpus at index
>> time you
>> could index these "similar" documents contiguously. T
>>
>
> Wow.. that's shockingly frightening. There are a ton of optimizations if
> you can trick the underlying content store into performing locality.
>
> Not trying to be overly negative so another way to phrase it is that at
> least there's room for improvement !
>
>
>> My base question is why you'd care about compressing 500G. Disk space
>> is so cheap that the expense of trying to control this dwarfs any
>> imaginable
>> $avings, unless you're talking about a lot of 500G indexes. In other words
>> this seems like an
>> XY problem, you're asking about compressing when you are really concerned
>> with something else.
>>
>
> 500GB per day... additionally, disk is cheap, but IOPS are not. The more we
> can keep in ram and on SSD the better.
>
> And we're trying to get as much in RAM then SSD as possible... plus we have
> about 2 years of content. It adds up ;)
>
> Kevin
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Possible to cause documents to be contiguous after forceMerge?
Posted by Erick Erickson <er...@gmail.com>.
Well, codecs are pluggable so if you can show that you'd get
an improvement (however you measure them) and that whatever
you have in mind wouldn't penalize the general case you could
submit it as a proposal/patch.
Best,
Erick
On Tue, Nov 15, 2016 at 6:21 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> On Tue, Nov 15, 2016 at 6:16 PM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> You can make no assumptions about locality in terms of where separate
>> documents land on disk. I suppose if you have the whole corpus at index
>> time you
>> could index these "similar" documents contiguously. T
>>
>
> Wow.. that's shockingly frightening. There are a ton of optimizations if
> you can trick the underlying content store into performing locality.
>
> Not trying to be overly negative so another way to phrase it is that at
> least there's room for improvement !
>
>
>> My base question is why you'd care about compressing 500G. Disk space
>> is so cheap that the expense of trying to control this dwarfs any
>> imaginable
>> $avings, unless you're talking about a lot of 500G indexes. In other words
>> this seems like an
>> XY problem, you're asking about compressing when you are really concerned
>> with something else.
>>
>
> 500GB per day... additionally, disk is cheap, but IOPS are not. The more we
> can keep in ram and on SSD the better.
>
> And we're trying to get as much in RAM then SSD as possible... plus we have
> about 2 years of content. It adds up ;)
>
> Kevin
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Possible to cause documents to be contiguous after forceMerge?
Posted by Kevin Burton <bu...@spinn3r.com>.
On Tue, Nov 15, 2016 at 6:16 PM, Erick Erickson <er...@gmail.com>
wrote:
> You can make no assumptions about locality in terms of where separate
> documents land on disk. I suppose if you have the whole corpus at index
> time you
> could index these "similar" documents contiguously. T
>
Wow.. that's shockingly frightening. There are a ton of optimizations if
you can trick the underlying content store into performing locality.
Not trying to be overly negative so another way to phrase it is that at
least there's room for improvement !
> My base question is why you'd care about compressing 500G. Disk space
> is so cheap that the expense of trying to control this dwarfs any
> imaginable
> $avings, unless you're talking about a lot of 500G indexes. In other words
> this seems like an
> XY problem, you're asking about compressing when you are really concerned
> with something else.
>
500GB per day... additionally, disk is cheap, but IOPS are not. The more we
can keep in ram and on SSD the better.
And we're trying to get as much in RAM then SSD as possible... plus we have
about 2 years of content. It adds up ;)
Kevin
--
We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Possible to cause documents to be contiguous after forceMerge?
Posted by Erick Erickson <er...@gmail.com>.
You can make no assumptions about locality in terms of where separate
documents land
on disk. I suppose if you have the whole corpus at index time you
could index these
"similar" documents contiguously. Then, assuming there was absolutely never any
updates/deletes I _think_ the doc might tend to be contiguous on disk
but that's very iffy
and based on several
My base question is why you'd care about compressing 500G. Disk space
is so cheap
that the expense of trying to control this dwarfs any imaginable
$avings, unless you're
talking about a lot of 500G indexes. In other words this seems like an
XY problem, you're
asking about compressing when you are really concerned with something else.
Best,
Erick
On Tue, Nov 15, 2016 at 5:32 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> I have a large index (say 500GB) that with a large percentage of near
> duplicate documents.
>
> I have to keep the documents there (can't delete them) as the metadata is
> important.
>
> Is it possible to get the documents to be contiguous somehow?
>
> Once they are contiguous then they will compress very well - which I've
> already confirmed by writing the exact same document N times.
>
> IDEALLY I could use two fields and have a unique document ID but then a
> group_id so that they can be located on disk by the group_id... but I don't
> think this is possible.
>
> Can I just create a synthetic "id" field for this and assume that "id" is
> ordered on disk in the lucene index?
>
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org