You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Joe Zhang <sm...@gmail.com> on 2013/07/27 07:02:43 UTC
processing documents in solr
Dear list:
I have an ever-growing solr repository, and I need to process every single
document to extract statistics. What would be a reasonable process that
satifies the following properties:
- Exhaustive: I have to traverse every single document
- Incremental: in other words, it has to allow me to divide and conquer ---
if I have processed the first 20k docs, next time I can start with 20001.
A simple "*:*" query would satisfy the 1st but not the 2nd property. In
fact, given that the processing will take very long, and the repository
keeps growing, it is not even clear that the exhaustiveness is achieved.
I'm running solr 3.6.2 in a single-machine setting; no hadoop capability
yet. But I guess the same issues still hold even if I have the solr cloud
environment, right, say in each shard?
Any help would be greatly appreciated.
Joe
Re: processing documents in solr
Posted by Roman Chyla <ro...@gmail.com>.
On Sat, Jul 27, 2013 at 4:17 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 7/27/2013 11:38 AM, Joe Zhang wrote:
> > I have a constantly growing index, so not updating the index can't be
> > practical...
> >
> > Going back to the beginning of this thread: when we use the vanilla
> > "*:*"+pagination approach, would the ordering of documents remain stable?
> > ---- the index is dynamic: update/insertion only, no deletion.
>
> If you use a sort parameter with pagination, then you have stable
> ordering, unless as described with the 'b' example, a new document gets
> inserted into a position in the sort sequence that's before the current
> result page.
>
> One thing that you could do is make a copy of your index, set up a
> separate Solr installation that's not getting updates, and use that for
> your inspection.
>
Hi Shawn,
I guess if something prevents the current searcher from being recycled
(e.g. incrementing its ref count), it would be possible to re-use it for
the pagination - then the consumer could get tight to the reader and the
order is stable (seeing the same data) but there probably is not a
mechanism for this (?) nor would it be very wise to have such a mechanism
(?).
roman
>
> Thanks,
> Shawn
>
>
Re: processing documents in solr
Posted by Shawn Heisey <so...@elyograg.org>.
On 7/27/2013 11:38 AM, Joe Zhang wrote:
> I have a constantly growing index, so not updating the index can't be
> practical...
>
> Going back to the beginning of this thread: when we use the vanilla
> "*:*"+pagination approach, would the ordering of documents remain stable?
> ---- the index is dynamic: update/insertion only, no deletion.
If you use a sort parameter with pagination, then you have stable
ordering, unless as described with the 'b' example, a new document gets
inserted into a position in the sort sequence that's before the current
result page.
One thing that you could do is make a copy of your index, set up a
separate Solr installation that's not getting updates, and use that for
your inspection.
Thanks,
Shawn
Re: processing documents in solr
Posted by Joe Zhang <sm...@gmail.com>.
I have a constantly growing index, so not updating the index can't be
practical...
Going back to the beginning of this thread: when we use the vanilla
"*:*"+pagination approach, would the ordering of documents remain stable?
---- the index is dynamic: update/insertion only, no deletion.
On Sat, Jul 27, 2013 at 10:28 AM, Shawn Heisey <so...@elyograg.org> wrote:
> On 7/27/2013 11:17 AM, Joe Zhang wrote:
> > Thanks for sharing, Roman. I'll look into your code.
> >
> > One more thought on your suggestion, Shawn. In fact, for the id, we need
> > more than "unique" and "rangeable"; we also need some sense of atomic
> > values. Your approach might run into risk with a text-based id field,
> say:
> >
> > the id/key has values 'a', 'c', 'f', 'g', and our pagesize is 2. Your
> > suggestion would work fine. But with newly added documents, there is no
> > guarantee that they are not going to use the key value 'b'. And this new
> > document would be missed in your algorithm, right?
>
> That's why I said that you would either have to not update the index or
> ensure that (in your example) a 'b' document never gets added. Because
> you can't make that kind of guarantee in most situations, not updating
> the index is safer.
>
> Thanks,
> Shawn
>
>
Re: processing documents in solr
Posted by Shawn Heisey <so...@elyograg.org>.
On 7/27/2013 11:17 AM, Joe Zhang wrote:
> Thanks for sharing, Roman. I'll look into your code.
>
> One more thought on your suggestion, Shawn. In fact, for the id, we need
> more than "unique" and "rangeable"; we also need some sense of atomic
> values. Your approach might run into risk with a text-based id field, say:
>
> the id/key has values 'a', 'c', 'f', 'g', and our pagesize is 2. Your
> suggestion would work fine. But with newly added documents, there is no
> guarantee that they are not going to use the key value 'b'. And this new
> document would be missed in your algorithm, right?
That's why I said that you would either have to not update the index or
ensure that (in your example) a 'b' document never gets added. Because
you can't make that kind of guarantee in most situations, not updating
the index is safer.
Thanks,
Shawn
Re: processing documents in solr
Posted by Joe Zhang <sm...@gmail.com>.
Thanks for sharing, Roman. I'll look into your code.
One more thought on your suggestion, Shawn. In fact, for the id, we need
more than "unique" and "rangeable"; we also need some sense of atomic
values. Your approach might run into risk with a text-based id field, say:
the id/key has values 'a', 'c', 'f', 'g', and our pagesize is 2. Your
suggestion would work fine. But with newly added documents, there is no
guarantee that they are not going to use the key value 'b'. And this new
document would be missed in your algorithm, right?
On Sat, Jul 27, 2013 at 5:32 AM, Roman Chyla <ro...@gmail.com> wrote:
> Dear list,
> I'vw written a special processor exactly for this kind of operations
>
>
> https://github.com/romanchyla/montysolr/tree/master/contrib/adsabs/src/java/org/apache/solr/handler/batch
>
> This is how we use it
> http://labs.adsabs.harvard.edu/trac/ads-invenio/wiki/SearchEngineBatch
>
> It is capable of processing index of 200gb in few minutes,
> copying/streaming large amounts of data is normal
>
> If there is general interest, we can create jira issue - but given my
> current workload time, it will take longer and also somebody else will
> *have to* invest their time and energy in testing it, reporting, etc. Of
> course, feel free to create the jira yourself or reuse the code -
> hopefully, you will improve it and let me know ;-)
>
> Roman
> On 27 Jul 2013 01:03, "Joe Zhang" <sm...@gmail.com> wrote:
>
> > Dear list:
> >
> > I have an ever-growing solr repository, and I need to process every
> single
> > document to extract statistics. What would be a reasonable process that
> > satifies the following properties:
> >
> > - Exhaustive: I have to traverse every single document
> > - Incremental: in other words, it has to allow me to divide and conquer
> ---
> > if I have processed the first 20k docs, next time I can start with 20001.
> >
> > A simple "*:*" query would satisfy the 1st but not the 2nd property. In
> > fact, given that the processing will take very long, and the repository
> > keeps growing, it is not even clear that the exhaustiveness is achieved.
> >
> > I'm running solr 3.6.2 in a single-machine setting; no hadoop capability
> > yet. But I guess the same issues still hold even if I have the solr cloud
> > environment, right, say in each shard?
> >
> > Any help would be greatly appreciated.
> >
> > Joe
> >
>
Re: processing documents in solr
Posted by Roman Chyla <ro...@gmail.com>.
Dear list,
I'vw written a special processor exactly for this kind of operations
https://github.com/romanchyla/montysolr/tree/master/contrib/adsabs/src/java/org/apache/solr/handler/batch
This is how we use it
http://labs.adsabs.harvard.edu/trac/ads-invenio/wiki/SearchEngineBatch
It is capable of processing index of 200gb in few minutes,
copying/streaming large amounts of data is normal
If there is general interest, we can create jira issue - but given my
current workload time, it will take longer and also somebody else will
*have to* invest their time and energy in testing it, reporting, etc. Of
course, feel free to create the jira yourself or reuse the code -
hopefully, you will improve it and let me know ;-)
Roman
On 27 Jul 2013 01:03, "Joe Zhang" <sm...@gmail.com> wrote:
> Dear list:
>
> I have an ever-growing solr repository, and I need to process every single
> document to extract statistics. What would be a reasonable process that
> satifies the following properties:
>
> - Exhaustive: I have to traverse every single document
> - Incremental: in other words, it has to allow me to divide and conquer ---
> if I have processed the first 20k docs, next time I can start with 20001.
>
> A simple "*:*" query would satisfy the 1st but not the 2nd property. In
> fact, given that the processing will take very long, and the repository
> keeps growing, it is not even clear that the exhaustiveness is achieved.
>
> I'm running solr 3.6.2 in a single-machine setting; no hadoop capability
> yet. But I guess the same issues still hold even if I have the solr cloud
> environment, right, say in each shard?
>
> Any help would be greatly appreciated.
>
> Joe
>
Re: processing documents in solr
Posted by Joe Zhang <sm...@gmail.com>.
I'll try reindexing the timestamp.
The id-creation approach suggested by Erick sounds attractive, but the
nutch/solr integration seems rather tight. I don't where to break in to
insert the id into solr.
On Mon, Jul 29, 2013 at 4:11 AM, Erick Erickson <er...@gmail.com>wrote:
> No SolrJ doesn't provide this automatically. You'd be providing the
> counter by inserting it into the document as you created new docs.
>
> You could do this with any kind of document creation you are
> using.
>
> Best
> Erick
>
> On Mon, Jul 29, 2013 at 2:51 AM, Aditya <fi...@gmail.com>
> wrote:
> > Hi,
> >
> > The easiest solution would be to have timestamp indexed. Is there any
> issue
> > in doing re-indexing?
> > If you want to process records in batch then you need a ordered list and
> a
> > bookmark. You require a field to sort and maintain a counter / last id as
> > bookmark. This is mandatory to solve your problem.
> >
> > If you don't want to re-index, then you need to maintain information
> > related to visited nodes. Have a database / solr core which maintains
> list
> > of IDs which already processed. Fetch record from Solr, For each record,
> > check the new DB, if the record is already processed.
> >
> > Regards
> > Aditya
> > www.findbestopensource.com
> >
> >
> >
> >
> >
> > On Mon, Jul 29, 2013 at 10:26 AM, Joe Zhang <sm...@gmail.com>
> wrote:
> >
> >> Basically, I was thinking about running a range query like Shawn
> suggested
> >> on the tstamp field, but unfortunately it was not indexed. Range queries
> >> only work on indexed fields, right?
> >>
> >>
> >> On Sun, Jul 28, 2013 at 9:49 PM, Joe Zhang <sm...@gmail.com>
> wrote:
> >>
> >> > I've been thinking about tstamp solution int the past few days. but
> too
> >> > bad, the field is avaialble but not indexed...
> >> >
> >> > I'm not familiar with SolrJ. Again, sounds like SolrJ is providing the
> >> > counter value. If yes, that would be equivalent to an autoincrement
> id.
> >> I'm
> >> > indexing from Nutch though; don't know how to feed in such counter...
> >> >
> >> >
> >> > On Sun, Jul 28, 2013 at 7:03 AM, Erick Erickson <
> erickerickson@gmail.com
> >> >wrote:
> >> >
> >> >> Why wouldn't a simple timestamp work for the ordering? Although
> >> >> I guess "simple timestamp" isn't really simple if the time settings
> >> >> change.
> >> >>
> >> >> So how about a simple counter field in your documents? Assuming
> >> >> you're indexing from SolrJ, your setup is to query q=*:*&sort=counter
> >> >> desc.
> >> >> Take the counter from the first document returned. Increment for
> >> >> each doc for the life of the indexing run. Now you've got, for all
> >> intents
> >> >> and purposes, an identity field albeit manually maintained.
> >> >>
> >> >> Then use your counter field as Shawn suggests for pulling all the
> >> >> data out.
> >> >>
> >> >> FWIW,
> >> >> Erick
> >> >>
> >> >> On Sun, Jul 28, 2013 at 1:01 AM, Maurizio Cucchiara
> >> >> <mc...@apache.org> wrote:
> >> >> > In both cases, for better performance, first I'd load just all the
> >> IDs,
> >> >> > after, during processing I'd load each document.
> >> >> > For what concern the incremental requirement, it should not be
> >> >> difficult to
> >> >> > write an hash function which maps a non-numerical I'd to a value.
> >> >> > On Jul 27, 2013 7:03 AM, "Joe Zhang" <sm...@gmail.com>
> wrote:
> >> >> >
> >> >> >> Dear list:
> >> >> >>
> >> >> >> I have an ever-growing solr repository, and I need to process
> every
> >> >> single
> >> >> >> document to extract statistics. What would be a reasonable process
> >> that
> >> >> >> satifies the following properties:
> >> >> >>
> >> >> >> - Exhaustive: I have to traverse every single document
> >> >> >> - Incremental: in other words, it has to allow me to divide and
> >> >> conquer ---
> >> >> >> if I have processed the first 20k docs, next time I can start with
> >> >> 20001.
> >> >> >>
> >> >> >> A simple "*:*" query would satisfy the 1st but not the 2nd
> property.
> >> In
> >> >> >> fact, given that the processing will take very long, and the
> >> repository
> >> >> >> keeps growing, it is not even clear that the exhaustiveness is
> >> >> achieved.
> >> >> >>
> >> >> >> I'm running solr 3.6.2 in a single-machine setting; no hadoop
> >> >> capability
> >> >> >> yet. But I guess the same issues still hold even if I have the
> solr
> >> >> cloud
> >> >> >> environment, right, say in each shard?
> >> >> >>
> >> >> >> Any help would be greatly appreciated.
> >> >> >>
> >> >> >> Joe
> >> >> >>
> >> >>
> >> >
> >> >
> >>
>
Re: processing documents in solr
Posted by Erick Erickson <er...@gmail.com>.
No SolrJ doesn't provide this automatically. You'd be providing the
counter by inserting it into the document as you created new docs.
You could do this with any kind of document creation you are
using.
Best
Erick
On Mon, Jul 29, 2013 at 2:51 AM, Aditya <fi...@gmail.com> wrote:
> Hi,
>
> The easiest solution would be to have timestamp indexed. Is there any issue
> in doing re-indexing?
> If you want to process records in batch then you need a ordered list and a
> bookmark. You require a field to sort and maintain a counter / last id as
> bookmark. This is mandatory to solve your problem.
>
> If you don't want to re-index, then you need to maintain information
> related to visited nodes. Have a database / solr core which maintains list
> of IDs which already processed. Fetch record from Solr, For each record,
> check the new DB, if the record is already processed.
>
> Regards
> Aditya
> www.findbestopensource.com
>
>
>
>
>
> On Mon, Jul 29, 2013 at 10:26 AM, Joe Zhang <sm...@gmail.com> wrote:
>
>> Basically, I was thinking about running a range query like Shawn suggested
>> on the tstamp field, but unfortunately it was not indexed. Range queries
>> only work on indexed fields, right?
>>
>>
>> On Sun, Jul 28, 2013 at 9:49 PM, Joe Zhang <sm...@gmail.com> wrote:
>>
>> > I've been thinking about tstamp solution int the past few days. but too
>> > bad, the field is avaialble but not indexed...
>> >
>> > I'm not familiar with SolrJ. Again, sounds like SolrJ is providing the
>> > counter value. If yes, that would be equivalent to an autoincrement id.
>> I'm
>> > indexing from Nutch though; don't know how to feed in such counter...
>> >
>> >
>> > On Sun, Jul 28, 2013 at 7:03 AM, Erick Erickson <erickerickson@gmail.com
>> >wrote:
>> >
>> >> Why wouldn't a simple timestamp work for the ordering? Although
>> >> I guess "simple timestamp" isn't really simple if the time settings
>> >> change.
>> >>
>> >> So how about a simple counter field in your documents? Assuming
>> >> you're indexing from SolrJ, your setup is to query q=*:*&sort=counter
>> >> desc.
>> >> Take the counter from the first document returned. Increment for
>> >> each doc for the life of the indexing run. Now you've got, for all
>> intents
>> >> and purposes, an identity field albeit manually maintained.
>> >>
>> >> Then use your counter field as Shawn suggests for pulling all the
>> >> data out.
>> >>
>> >> FWIW,
>> >> Erick
>> >>
>> >> On Sun, Jul 28, 2013 at 1:01 AM, Maurizio Cucchiara
>> >> <mc...@apache.org> wrote:
>> >> > In both cases, for better performance, first I'd load just all the
>> IDs,
>> >> > after, during processing I'd load each document.
>> >> > For what concern the incremental requirement, it should not be
>> >> difficult to
>> >> > write an hash function which maps a non-numerical I'd to a value.
>> >> > On Jul 27, 2013 7:03 AM, "Joe Zhang" <sm...@gmail.com> wrote:
>> >> >
>> >> >> Dear list:
>> >> >>
>> >> >> I have an ever-growing solr repository, and I need to process every
>> >> single
>> >> >> document to extract statistics. What would be a reasonable process
>> that
>> >> >> satifies the following properties:
>> >> >>
>> >> >> - Exhaustive: I have to traverse every single document
>> >> >> - Incremental: in other words, it has to allow me to divide and
>> >> conquer ---
>> >> >> if I have processed the first 20k docs, next time I can start with
>> >> 20001.
>> >> >>
>> >> >> A simple "*:*" query would satisfy the 1st but not the 2nd property.
>> In
>> >> >> fact, given that the processing will take very long, and the
>> repository
>> >> >> keeps growing, it is not even clear that the exhaustiveness is
>> >> achieved.
>> >> >>
>> >> >> I'm running solr 3.6.2 in a single-machine setting; no hadoop
>> >> capability
>> >> >> yet. But I guess the same issues still hold even if I have the solr
>> >> cloud
>> >> >> environment, right, say in each shard?
>> >> >>
>> >> >> Any help would be greatly appreciated.
>> >> >>
>> >> >> Joe
>> >> >>
>> >>
>> >
>> >
>>
Re: processing documents in solr
Posted by Aditya <fi...@gmail.com>.
Hi,
The easiest solution would be to have timestamp indexed. Is there any issue
in doing re-indexing?
If you want to process records in batch then you need a ordered list and a
bookmark. You require a field to sort and maintain a counter / last id as
bookmark. This is mandatory to solve your problem.
If you don't want to re-index, then you need to maintain information
related to visited nodes. Have a database / solr core which maintains list
of IDs which already processed. Fetch record from Solr, For each record,
check the new DB, if the record is already processed.
Regards
Aditya
www.findbestopensource.com
On Mon, Jul 29, 2013 at 10:26 AM, Joe Zhang <sm...@gmail.com> wrote:
> Basically, I was thinking about running a range query like Shawn suggested
> on the tstamp field, but unfortunately it was not indexed. Range queries
> only work on indexed fields, right?
>
>
> On Sun, Jul 28, 2013 at 9:49 PM, Joe Zhang <sm...@gmail.com> wrote:
>
> > I've been thinking about tstamp solution int the past few days. but too
> > bad, the field is avaialble but not indexed...
> >
> > I'm not familiar with SolrJ. Again, sounds like SolrJ is providing the
> > counter value. If yes, that would be equivalent to an autoincrement id.
> I'm
> > indexing from Nutch though; don't know how to feed in such counter...
> >
> >
> > On Sun, Jul 28, 2013 at 7:03 AM, Erick Erickson <erickerickson@gmail.com
> >wrote:
> >
> >> Why wouldn't a simple timestamp work for the ordering? Although
> >> I guess "simple timestamp" isn't really simple if the time settings
> >> change.
> >>
> >> So how about a simple counter field in your documents? Assuming
> >> you're indexing from SolrJ, your setup is to query q=*:*&sort=counter
> >> desc.
> >> Take the counter from the first document returned. Increment for
> >> each doc for the life of the indexing run. Now you've got, for all
> intents
> >> and purposes, an identity field albeit manually maintained.
> >>
> >> Then use your counter field as Shawn suggests for pulling all the
> >> data out.
> >>
> >> FWIW,
> >> Erick
> >>
> >> On Sun, Jul 28, 2013 at 1:01 AM, Maurizio Cucchiara
> >> <mc...@apache.org> wrote:
> >> > In both cases, for better performance, first I'd load just all the
> IDs,
> >> > after, during processing I'd load each document.
> >> > For what concern the incremental requirement, it should not be
> >> difficult to
> >> > write an hash function which maps a non-numerical I'd to a value.
> >> > On Jul 27, 2013 7:03 AM, "Joe Zhang" <sm...@gmail.com> wrote:
> >> >
> >> >> Dear list:
> >> >>
> >> >> I have an ever-growing solr repository, and I need to process every
> >> single
> >> >> document to extract statistics. What would be a reasonable process
> that
> >> >> satifies the following properties:
> >> >>
> >> >> - Exhaustive: I have to traverse every single document
> >> >> - Incremental: in other words, it has to allow me to divide and
> >> conquer ---
> >> >> if I have processed the first 20k docs, next time I can start with
> >> 20001.
> >> >>
> >> >> A simple "*:*" query would satisfy the 1st but not the 2nd property.
> In
> >> >> fact, given that the processing will take very long, and the
> repository
> >> >> keeps growing, it is not even clear that the exhaustiveness is
> >> achieved.
> >> >>
> >> >> I'm running solr 3.6.2 in a single-machine setting; no hadoop
> >> capability
> >> >> yet. But I guess the same issues still hold even if I have the solr
> >> cloud
> >> >> environment, right, say in each shard?
> >> >>
> >> >> Any help would be greatly appreciated.
> >> >>
> >> >> Joe
> >> >>
> >>
> >
> >
>
Re: processing documents in solr
Posted by Joe Zhang <sm...@gmail.com>.
Basically, I was thinking about running a range query like Shawn suggested
on the tstamp field, but unfortunately it was not indexed. Range queries
only work on indexed fields, right?
On Sun, Jul 28, 2013 at 9:49 PM, Joe Zhang <sm...@gmail.com> wrote:
> I've been thinking about tstamp solution int the past few days. but too
> bad, the field is avaialble but not indexed...
>
> I'm not familiar with SolrJ. Again, sounds like SolrJ is providing the
> counter value. If yes, that would be equivalent to an autoincrement id. I'm
> indexing from Nutch though; don't know how to feed in such counter...
>
>
> On Sun, Jul 28, 2013 at 7:03 AM, Erick Erickson <er...@gmail.com>wrote:
>
>> Why wouldn't a simple timestamp work for the ordering? Although
>> I guess "simple timestamp" isn't really simple if the time settings
>> change.
>>
>> So how about a simple counter field in your documents? Assuming
>> you're indexing from SolrJ, your setup is to query q=*:*&sort=counter
>> desc.
>> Take the counter from the first document returned. Increment for
>> each doc for the life of the indexing run. Now you've got, for all intents
>> and purposes, an identity field albeit manually maintained.
>>
>> Then use your counter field as Shawn suggests for pulling all the
>> data out.
>>
>> FWIW,
>> Erick
>>
>> On Sun, Jul 28, 2013 at 1:01 AM, Maurizio Cucchiara
>> <mc...@apache.org> wrote:
>> > In both cases, for better performance, first I'd load just all the IDs,
>> > after, during processing I'd load each document.
>> > For what concern the incremental requirement, it should not be
>> difficult to
>> > write an hash function which maps a non-numerical I'd to a value.
>> > On Jul 27, 2013 7:03 AM, "Joe Zhang" <sm...@gmail.com> wrote:
>> >
>> >> Dear list:
>> >>
>> >> I have an ever-growing solr repository, and I need to process every
>> single
>> >> document to extract statistics. What would be a reasonable process that
>> >> satifies the following properties:
>> >>
>> >> - Exhaustive: I have to traverse every single document
>> >> - Incremental: in other words, it has to allow me to divide and
>> conquer ---
>> >> if I have processed the first 20k docs, next time I can start with
>> 20001.
>> >>
>> >> A simple "*:*" query would satisfy the 1st but not the 2nd property. In
>> >> fact, given that the processing will take very long, and the repository
>> >> keeps growing, it is not even clear that the exhaustiveness is
>> achieved.
>> >>
>> >> I'm running solr 3.6.2 in a single-machine setting; no hadoop
>> capability
>> >> yet. But I guess the same issues still hold even if I have the solr
>> cloud
>> >> environment, right, say in each shard?
>> >>
>> >> Any help would be greatly appreciated.
>> >>
>> >> Joe
>> >>
>>
>
>
Re: processing documents in solr
Posted by Joe Zhang <sm...@gmail.com>.
I've been thinking about tstamp solution int the past few days. but too
bad, the field is avaialble but not indexed...
I'm not familiar with SolrJ. Again, sounds like SolrJ is providing the
counter value. If yes, that would be equivalent to an autoincrement id. I'm
indexing from Nutch though; don't know how to feed in such counter...
On Sun, Jul 28, 2013 at 7:03 AM, Erick Erickson <er...@gmail.com>wrote:
> Why wouldn't a simple timestamp work for the ordering? Although
> I guess "simple timestamp" isn't really simple if the time settings
> change.
>
> So how about a simple counter field in your documents? Assuming
> you're indexing from SolrJ, your setup is to query q=*:*&sort=counter desc.
> Take the counter from the first document returned. Increment for
> each doc for the life of the indexing run. Now you've got, for all intents
> and purposes, an identity field albeit manually maintained.
>
> Then use your counter field as Shawn suggests for pulling all the
> data out.
>
> FWIW,
> Erick
>
> On Sun, Jul 28, 2013 at 1:01 AM, Maurizio Cucchiara
> <mc...@apache.org> wrote:
> > In both cases, for better performance, first I'd load just all the IDs,
> > after, during processing I'd load each document.
> > For what concern the incremental requirement, it should not be difficult
> to
> > write an hash function which maps a non-numerical I'd to a value.
> > On Jul 27, 2013 7:03 AM, "Joe Zhang" <sm...@gmail.com> wrote:
> >
> >> Dear list:
> >>
> >> I have an ever-growing solr repository, and I need to process every
> single
> >> document to extract statistics. What would be a reasonable process that
> >> satifies the following properties:
> >>
> >> - Exhaustive: I have to traverse every single document
> >> - Incremental: in other words, it has to allow me to divide and conquer
> ---
> >> if I have processed the first 20k docs, next time I can start with
> 20001.
> >>
> >> A simple "*:*" query would satisfy the 1st but not the 2nd property. In
> >> fact, given that the processing will take very long, and the repository
> >> keeps growing, it is not even clear that the exhaustiveness is achieved.
> >>
> >> I'm running solr 3.6.2 in a single-machine setting; no hadoop capability
> >> yet. But I guess the same issues still hold even if I have the solr
> cloud
> >> environment, right, say in each shard?
> >>
> >> Any help would be greatly appreciated.
> >>
> >> Joe
> >>
>
Re: processing documents in solr
Posted by Erick Erickson <er...@gmail.com>.
Why wouldn't a simple timestamp work for the ordering? Although
I guess "simple timestamp" isn't really simple if the time settings
change.
So how about a simple counter field in your documents? Assuming
you're indexing from SolrJ, your setup is to query q=*:*&sort=counter desc.
Take the counter from the first document returned. Increment for
each doc for the life of the indexing run. Now you've got, for all intents
and purposes, an identity field albeit manually maintained.
Then use your counter field as Shawn suggests for pulling all the
data out.
FWIW,
Erick
On Sun, Jul 28, 2013 at 1:01 AM, Maurizio Cucchiara
<mc...@apache.org> wrote:
> In both cases, for better performance, first I'd load just all the IDs,
> after, during processing I'd load each document.
> For what concern the incremental requirement, it should not be difficult to
> write an hash function which maps a non-numerical I'd to a value.
> On Jul 27, 2013 7:03 AM, "Joe Zhang" <sm...@gmail.com> wrote:
>
>> Dear list:
>>
>> I have an ever-growing solr repository, and I need to process every single
>> document to extract statistics. What would be a reasonable process that
>> satifies the following properties:
>>
>> - Exhaustive: I have to traverse every single document
>> - Incremental: in other words, it has to allow me to divide and conquer ---
>> if I have processed the first 20k docs, next time I can start with 20001.
>>
>> A simple "*:*" query would satisfy the 1st but not the 2nd property. In
>> fact, given that the processing will take very long, and the repository
>> keeps growing, it is not even clear that the exhaustiveness is achieved.
>>
>> I'm running solr 3.6.2 in a single-machine setting; no hadoop capability
>> yet. But I guess the same issues still hold even if I have the solr cloud
>> environment, right, say in each shard?
>>
>> Any help would be greatly appreciated.
>>
>> Joe
>>
Re: processing documents in solr
Posted by Maurizio Cucchiara <mc...@apache.org>.
In both cases, for better performance, first I'd load just all the IDs,
after, during processing I'd load each document.
For what concern the incremental requirement, it should not be difficult to
write an hash function which maps a non-numerical I'd to a value.
On Jul 27, 2013 7:03 AM, "Joe Zhang" <sm...@gmail.com> wrote:
> Dear list:
>
> I have an ever-growing solr repository, and I need to process every single
> document to extract statistics. What would be a reasonable process that
> satifies the following properties:
>
> - Exhaustive: I have to traverse every single document
> - Incremental: in other words, it has to allow me to divide and conquer ---
> if I have processed the first 20k docs, next time I can start with 20001.
>
> A simple "*:*" query would satisfy the 1st but not the 2nd property. In
> fact, given that the processing will take very long, and the repository
> keeps growing, it is not even clear that the exhaustiveness is achieved.
>
> I'm running solr 3.6.2 in a single-machine setting; no hadoop capability
> yet. But I guess the same issues still hold even if I have the solr cloud
> environment, right, say in each shard?
>
> Any help would be greatly appreciated.
>
> Joe
>
Re: processing documents in solr
Posted by Joe Zhang <sm...@gmail.com>.
On a related, inspired by what you said, Shawn, an auto increment id seems
perfect here. Yet I found there is no such support in solr. The UUID only
guarantees uniqueness.
On Fri, Jul 26, 2013 at 10:50 PM, Joe Zhang <sm...@gmail.com> wrote:
> Thanks for your kind reply, Shawn.
>
> On Fri, Jul 26, 2013 at 10:27 PM, Shawn Heisey <so...@elyograg.org> wrote:
>
>> On 7/26/2013 11:02 PM, Joe Zhang wrote:
>> > I have an ever-growing solr repository, and I need to process every
>> single
>> > document to extract statistics. What would be a reasonable process that
>> > satifies the following properties:
>> >
>> > - Exhaustive: I have to traverse every single document
>> > - Incremental: in other words, it has to allow me to divide and conquer
>> ---
>> > if I have processed the first 20k docs, next time I can start with
>> 20001.
>>
>> If your index isn't very big, a *:* query with rows and start parameters
>> is perfectly acceptable. Performance is terrible for this method when
>> the index gets huge, though.
>>
>
> ==> Essentially we are doing paigination here, right? If performance is
> not the concern, given that the index is dynamic, does the order of
> entries remain stable over time?
>
>
>
>> If "id" is your uniqueKey field, here's how you can do it. If that's
>> not your uniqueKey field, substitute your uniqueKey field for id. This
>> method doesn't work properly if you don't use a field with values that
>> are guaranteed to be unique.
>>
>> For the first query, send a query with these parameters, where NNNNNN is
>> the number of docs you want to retrieve at once:
>> q=*:*&rows=NNNNNN&sort=id asc
>>
>> For each subsequent query, use the following parameters, where XXX is
>> the highest id value seen in the previous query:
>> q={XXX TO *}&rows=NNNNNN&sort=id asc
>>
>> ==> This approach seems to require that the id field is numerical, right?
> I have a text-based id that is unique.
>
> ==> I'm not sure I understand the "q={XXX TO *}" part --> wouldn't query
> be matched against the default search field, which could be "content", for
> example? How would that do the job?
>
>
>> As soon as you see a numFound value less than NNNNNN, you will know that
>> there's no more data.
>>
>> Generally speaking, you'd want to avoid updating the index while doing
>> these queries. If you never replace existing documents and you can
>> guarantee that the value in the uniqueKey field for new documents will
>> always be higher than any previous value, then you could continue
>> updating the index. A database autoincrement field would qualify for
>> that condition.
>>
>> Thanks,
>> Shawn
>>
>>
>
Re: processing documents in solr
Posted by Joe Zhang <sm...@gmail.com>.
Thanks.
On Fri, Jul 26, 2013 at 11:34 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 7/27/2013 12:30 AM, Joe Zhang wrote:
> > ==> so a "url" field would work fine?
>
> As long as it's guaranteed unique on every document (especially if it is
> your uniqueKey) and goes into the index as a single token, that should
> work just fine for the range queries I've described.
>
> Thanks,
> Shawn
>
>
Re: processing documents in solr
Posted by Shawn Heisey <so...@elyograg.org>.
On 7/27/2013 12:30 AM, Joe Zhang wrote:
> ==> so a "url" field would work fine?
As long as it's guaranteed unique on every document (especially if it is
your uniqueKey) and goes into the index as a single token, that should
work just fine for the range queries I've described.
Thanks,
Shawn
Re: processing documents in solr
Posted by Joe Zhang <sm...@gmail.com>.
On Fri, Jul 26, 2013 at 11:18 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 7/26/2013 11:50 PM, Joe Zhang wrote:
> > ==> Essentially we are doing paigination here, right? If performance is
> not
> > the concern, given that the index is dynamic, does the order of
> > entries remain stable over time?
>
> Yes, it's pagination. Just like the other method that I've described in
> detail, you'd have to avoid updating the index while you were getting
> information. Unless you can come up with a sort parameter that's
> guaranteed to make sure that new documents are at the end, any changes
> to the index during the retrieval process will make it impossible to
> retrieve every document.
>
==> What I can guarantee is that there is no deletion, but I guess this is
not equivalent to "newly added docs are at the end", right?
==> I believe you are right about performance. The retrived set becomes
larger and larger.
>
> >> ==> This approach seems to require that the id field is numerical,
> right?
> > I have a text-based id that is unique.
>
> StrField types work perfectly with range queries. As long as it's not a
> tokenized field, TextField works properly with range queries too.
> KeywordTokenizer is OK, as long you don't use filters that create
> additional tokens. Some examples that create additional tokens are
> WordDelimiterFilter and EdgeNgramFilter.
>
>
==> so a "url" field would work fine?
>
>
> ==> I'm not sure I understand the "q={XXX TO *}" part --> wouldn't query
> be
> > matched against the default search field, which could be "content", for
> > example? How would that do the job?
>
> You are correct, I was too hasty in constructing the query. That should
> be:
> q=id:{XXX TO *}&rows=NNNNNN&sort=id asc
>
> You could speed things up if you don't need to see all stored fields in
> the response by using the fl parameter to only return the fields that
> you need.
>
> Responding to your additional message about an autoincrement field -
> that would only be possible if you are importing from a data source that
> supports autoincrement, like MySQL. Solr itself has no support for
> autoincrement.
>
> Thanks,
> Shawn
>
>
Re: processing documents in solr
Posted by Shawn Heisey <so...@elyograg.org>.
On 7/26/2013 11:50 PM, Joe Zhang wrote:
> ==> Essentially we are doing paigination here, right? If performance is not
> the concern, given that the index is dynamic, does the order of
> entries remain stable over time?
Yes, it's pagination. Just like the other method that I've described in
detail, you'd have to avoid updating the index while you were getting
information. Unless you can come up with a sort parameter that's
guaranteed to make sure that new documents are at the end, any changes
to the index during the retrieval process will make it impossible to
retrieve every document.
>> ==> This approach seems to require that the id field is numerical, right?
> I have a text-based id that is unique.
StrField types work perfectly with range queries. As long as it's not a
tokenized field, TextField works properly with range queries too.
KeywordTokenizer is OK, as long you don't use filters that create
additional tokens. Some examples that create additional tokens are
WordDelimiterFilter and EdgeNgramFilter.
> ==> I'm not sure I understand the "q={XXX TO *}" part --> wouldn't query be
> matched against the default search field, which could be "content", for
> example? How would that do the job?
You are correct, I was too hasty in constructing the query. That should be:
q=id:{XXX TO *}&rows=NNNNNN&sort=id asc
You could speed things up if you don't need to see all stored fields in
the response by using the fl parameter to only return the fields that
you need.
Responding to your additional message about an autoincrement field -
that would only be possible if you are importing from a data source that
supports autoincrement, like MySQL. Solr itself has no support for
autoincrement.
Thanks,
Shawn
Re: processing documents in solr
Posted by Joe Zhang <sm...@gmail.com>.
Thanks for your kind reply, Shawn.
On Fri, Jul 26, 2013 at 10:27 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 7/26/2013 11:02 PM, Joe Zhang wrote:
> > I have an ever-growing solr repository, and I need to process every
> single
> > document to extract statistics. What would be a reasonable process that
> > satifies the following properties:
> >
> > - Exhaustive: I have to traverse every single document
> > - Incremental: in other words, it has to allow me to divide and conquer
> ---
> > if I have processed the first 20k docs, next time I can start with 20001.
>
> If your index isn't very big, a *:* query with rows and start parameters
> is perfectly acceptable. Performance is terrible for this method when
> the index gets huge, though.
>
==> Essentially we are doing paigination here, right? If performance is not
the concern, given that the index is dynamic, does the order of
entries remain stable over time?
> If "id" is your uniqueKey field, here's how you can do it. If that's
> not your uniqueKey field, substitute your uniqueKey field for id. This
> method doesn't work properly if you don't use a field with values that
> are guaranteed to be unique.
>
> For the first query, send a query with these parameters, where NNNNNN is
> the number of docs you want to retrieve at once:
> q=*:*&rows=NNNNNN&sort=id asc
>
> For each subsequent query, use the following parameters, where XXX is
> the highest id value seen in the previous query:
> q={XXX TO *}&rows=NNNNNN&sort=id asc
>
> ==> This approach seems to require that the id field is numerical, right?
I have a text-based id that is unique.
==> I'm not sure I understand the "q={XXX TO *}" part --> wouldn't query be
matched against the default search field, which could be "content", for
example? How would that do the job?
> As soon as you see a numFound value less than NNNNNN, you will know that
> there's no more data.
>
> Generally speaking, you'd want to avoid updating the index while doing
> these queries. If you never replace existing documents and you can
> guarantee that the value in the uniqueKey field for new documents will
> always be higher than any previous value, then you could continue
> updating the index. A database autoincrement field would qualify for
> that condition.
>
> Thanks,
> Shawn
>
>
Re: processing documents in solr
Posted by Shawn Heisey <so...@elyograg.org>.
On 7/26/2013 11:02 PM, Joe Zhang wrote:
> I have an ever-growing solr repository, and I need to process every single
> document to extract statistics. What would be a reasonable process that
> satifies the following properties:
>
> - Exhaustive: I have to traverse every single document
> - Incremental: in other words, it has to allow me to divide and conquer ---
> if I have processed the first 20k docs, next time I can start with 20001.
If your index isn't very big, a *:* query with rows and start parameters
is perfectly acceptable. Performance is terrible for this method when
the index gets huge, though.
If "id" is your uniqueKey field, here's how you can do it. If that's
not your uniqueKey field, substitute your uniqueKey field for id. This
method doesn't work properly if you don't use a field with values that
are guaranteed to be unique.
For the first query, send a query with these parameters, where NNNNNN is
the number of docs you want to retrieve at once:
q=*:*&rows=NNNNNN&sort=id asc
For each subsequent query, use the following parameters, where XXX is
the highest id value seen in the previous query:
q={XXX TO *}&rows=NNNNNN&sort=id asc
As soon as you see a numFound value less than NNNNNN, you will know that
there's no more data.
Generally speaking, you'd want to avoid updating the index while doing
these queries. If you never replace existing documents and you can
guarantee that the value in the uniqueKey field for new documents will
always be higher than any previous value, then you could continue
updating the index. A database autoincrement field would qualify for
that condition.
Thanks,
Shawn