You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mikhail Khludnev <mk...@griddynamics.com> on 2013/07/27 13:54:27 UTC

Re: paging vs streaming. spawn from (Processing a lot of results in Solr)

Otis,
You gave links to 'deep paging' when I asked about response streaming.
Let me understand. From my POV, deep paging is a special case for regular
search scenarios. We definitely need it in Solr. However, if we are talking
about data analytic like problems, when we need to select an "endless"
stream of responses (or store them in file as Roman did), 'deep paging' is
a suboptimal hack.
What's your vision on this?

Re: paging vs streaming. spawn from (Processing a lot of results in Solr)

Posted by Yonik Seeley <yo...@lucidworks.com>.
On Sat, Jul 27, 2013 at 4:30 PM, Roman Chyla <ro...@gmail.com> wrote:
> Let me just explain better what I found when I dug inside solr: documents
> (results of the query) are loaded before they are passed into a writer - so
> the writers are expecting to encounter the solr documents, but these
> documents were loaded by one of the components before rendering them

Hmmm, are you saying that it looks like documents are not being streamed?
Solr was designed to stream documents from a single server from day 1...
currently all that is collected up-front is the list of internal
docids (an int[]) and the stored fields are loaded and streamed back
one by one.

Of course it's certainly possible that someone introduced a bug, so we
should investigate if you think you see non-streaming action from a
single server.  Distributed is a different can of worms ;-)

-Yonik
http://lucidworks.com

Re: paging vs streaming. spawn from (Processing a lot of results in Solr)

Posted by Erick Erickson <er...@gmail.com>.
Shawn had an interesting idea on another thread. It depends
on having basically an identity field (which I see how to do
manually, but don't see how to make work as a new field type
in a distributed environment). And it's brilliantly simple, just
a range query identity:{NNNN TO *]&sort=identity asc.

Then just keep replacing the NNN with the max value you
got in the last packet. Of course this doesn't work if you
need to rank the results, but as a way to process all documents
in a corpus it seems to work. It's certainly not a general solution
to deep paging, but for the limited data dump case...

You could keep from processing the same doc twice (let's
say a doc gets updated and the identity field gets bumped)
by getting the min and max at the start of the dump.

But life is complicated. Siiiigggh. Doesn't work for M/R jobs
that compose an index from pieces either.

FWIW,
Erick


On Sun, Jul 28, 2013 at 1:28 AM, Mikhail Khludnev
<mk...@griddynamics.com> wrote:
> On Sun, Jul 28, 2013 at 1:25 AM, Yonik Seeley <yo...@lucidworks.com> wrote:
>
>>
>> Which part is problematic... the creation of the DocList (the search),
>>
> Literally DocList is a copy of TopDocs. Creating TopDocs is not a search,
> but ranking.
> And ranking costs is log(rows+start) beside of numFound, which the search
> takes.
> Interesting that we still pay that log() even if ask for collecting docs
> as-is with _docid_
>
>
>> or it's memory requirements (an int per doc)?
>>
> TopXxxCollector as well as XxxComparators allocates same [rows+start]
>
> it's clear that after we have deep paging, we need to handle heaps just
> with size of rows (without start).
> It's fairly ok, if we use Solr like site navigation engine, but it's
> 'sub-optimal' for data analytic use-cases, where we need something like
> SELECT * FROM ... in rdbms. In this case any memory allocation on billions
> docs index is a bummer. That's why I'm asking about removing heap based
> collector/comparator.
>
>
>> -Yonik
>> http://lucidworks.com
>>
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
>  <mk...@griddynamics.com>

Re: paging vs streaming. spawn from (Processing a lot of results in Solr)

Posted by Mikhail Khludnev <mk...@griddynamics.com>.
On Sun, Jul 28, 2013 at 1:25 AM, Yonik Seeley <yo...@lucidworks.com> wrote:

>
> Which part is problematic... the creation of the DocList (the search),
>
Literally DocList is a copy of TopDocs. Creating TopDocs is not a search,
but ranking.
And ranking costs is log(rows+start) beside of numFound, which the search
takes.
Interesting that we still pay that log() even if ask for collecting docs
as-is with _docid_


> or it's memory requirements (an int per doc)?
>
TopXxxCollector as well as XxxComparators allocates same [rows+start]

it's clear that after we have deep paging, we need to handle heaps just
with size of rows (without start).
It's fairly ok, if we use Solr like site navigation engine, but it's
'sub-optimal' for data analytic use-cases, where we need something like
SELECT * FROM ... in rdbms. In this case any memory allocation on billions
docs index is a bummer. That's why I'm asking about removing heap based
collector/comparator.


> -Yonik
> http://lucidworks.com
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
 <mk...@griddynamics.com>

Re: paging vs streaming. spawn from (Processing a lot of results in Solr)

Posted by Yonik Seeley <yo...@lucidworks.com>.
On Sat, Jul 27, 2013 at 5:05 PM, Mikhail Khludnev
<mk...@griddynamics.com> wrote:
> anyway, even if writer pulls docs one by one, it doesn't allow to stream a
> billion of them. Solr writes out DocList, which is really problematic even
> in deep-paging scenarios.

Which part is problematic... the creation of the DocList (the search),
or it's memory requirements (an int per doc)?

-Yonik
http://lucidworks.com

Re: paging vs streaming. spawn from (Processing a lot of results in Solr)

Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Hello,

Please find below


> Let me just explain better what I found when I dug inside solr: documents
> (results of the query) are loaded before they are passed into a writer - so
> the writers are expecting to encounter the solr documents, but these
> documents were loaded by one of the components before rendering them - so
> it is kinda 'hard-coded'.

there is the code
https://github.com/apache/lucene-solr/blob/trunk/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java#L445which
pulls documents into document's cache
to achieve your goal you can try to remove documents cache, or disable lazy
fields loading.


> But if solr was NOT loading these docs before
> passing them to a writer, writer can load them instead (hence lazy loading,
> but the difference is in numbers - it could deal with hundreds of thousands
> of docs, instead of few thousands now).
>

anyway, even if writer pulls docs one by one, it doesn't allow to stream a
billion of them. Solr writes out DocList, which is really problematic even
in deep-paging scenarios.


>
>
> roman
>
>
> On Sat, Jul 27, 2013 at 3:52 PM, Mikhail Khludnev <
> mkhludnev@griddynamics.com> wrote:
>
> > Roman,
> >
> > Let me briefly explain  the design
> >
> > special RequestParser stores servlet output stream into the context
> > https://github.com/m-khl/solr-patches/compare/streaming#L7R22
> >
> > then special component injects special PostFilter/DelegatingCollector
> which
> > writes right into output
> > https://github.com/m-khl/solr-patches/compare/streaming#L2R146
> >
> > here is how it streams the doc, you see it's lazy enough
> > https://github.com/m-khl/solr-patches/compare/streaming#L2R181
> >
> > I mention that it disables later collectors
> > https://github.com/m-khl/solr-patches/compare/streaming#L2R57
> > hence, no facets with streaming, yet as well as memory consumption.
> >
> > This test shows how it works
> > https://github.com/m-khl/solr-patches/compare/streaming#L15R115
> >
> > all other code purposed for distributed search.
> >
> >
> >
> > On Sat, Jul 27, 2013 at 4:44 PM, Roman Chyla <ro...@gmail.com>
> > wrote:
> >
> > > Mikhail,
> > > If your solution gives lazy loading of solr docs /and thus streaming of
> > > huge result lists/ it should be big YES!
> > > Roman
> > > On 27 Jul 2013 07:55, "Mikhail Khludnev" <mk...@griddynamics.com>
> > > wrote:
> > >
> > > > Otis,
> > > > You gave links to 'deep paging' when I asked about response
> streaming.
> > > > Let me understand. From my POV, deep paging is a special case for
> > regular
> > > > search scenarios. We definitely need it in Solr. However, if we are
> > > talking
> > > > about data analytic like problems, when we need to select an
> "endless"
> > > > stream of responses (or store them in file as Roman did), 'deep
> paging'
> > > is
> > > > a suboptimal hack.
> > > > What's your vision on this?
> > > >
> > >
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > Principal Engineer,
> > Grid Dynamics
> >
> > <http://www.griddynamics.com>
> >  <mk...@griddynamics.com>
> >
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
 <mk...@griddynamics.com>

Re: paging vs streaming. spawn from (Processing a lot of results in Solr)

Posted by Roman Chyla <ro...@gmail.com>.
Hi Mikhail,

I can see it is lazy-loading, but I can't judge how much complex it becomes
(presumably, the filter dispatching mechanism is doing also other things -
it is there not only for streaming).

Let me just explain better what I found when I dug inside solr: documents
(results of the query) are loaded before they are passed into a writer - so
the writers are expecting to encounter the solr documents, but these
documents were loaded by one of the components before rendering them - so
it is kinda 'hard-coded'. But if solr was NOT loading these docs before
passing them to a writer, writer can load them instead (hence lazy loading,
but the difference is in numbers - it could deal with hundreds of thousands
of docs, instead of few thousands now).

I see one crucial point: this could work without any new handler/servlet -
solr would just gain a new parameter, something like: 'lazy=true' ;) and
people can use whatever 'wt' they did before

disclaimer: i don't know whether that would break other stuff, I only know
that I am using the same idea to dump what i need without breaking things
(so far...;-)) - but obviously, i didn't want to patch solr core

roman


On Sat, Jul 27, 2013 at 3:52 PM, Mikhail Khludnev <
mkhludnev@griddynamics.com> wrote:

> Roman,
>
> Let me briefly explain  the design
>
> special RequestParser stores servlet output stream into the context
> https://github.com/m-khl/solr-patches/compare/streaming#L7R22
>
> then special component injects special PostFilter/DelegatingCollector which
> writes right into output
> https://github.com/m-khl/solr-patches/compare/streaming#L2R146
>
> here is how it streams the doc, you see it's lazy enough
> https://github.com/m-khl/solr-patches/compare/streaming#L2R181
>
> I mention that it disables later collectors
> https://github.com/m-khl/solr-patches/compare/streaming#L2R57
> hence, no facets with streaming, yet as well as memory consumption.
>
> This test shows how it works
> https://github.com/m-khl/solr-patches/compare/streaming#L15R115
>
> all other code purposed for distributed search.
>
>
>
> On Sat, Jul 27, 2013 at 4:44 PM, Roman Chyla <ro...@gmail.com>
> wrote:
>
> > Mikhail,
> > If your solution gives lazy loading of solr docs /and thus streaming of
> > huge result lists/ it should be big YES!
> > Roman
> > On 27 Jul 2013 07:55, "Mikhail Khludnev" <mk...@griddynamics.com>
> > wrote:
> >
> > > Otis,
> > > You gave links to 'deep paging' when I asked about response streaming.
> > > Let me understand. From my POV, deep paging is a special case for
> regular
> > > search scenarios. We definitely need it in Solr. However, if we are
> > talking
> > > about data analytic like problems, when we need to select an "endless"
> > > stream of responses (or store them in file as Roman did), 'deep paging'
> > is
> > > a suboptimal hack.
> > > What's your vision on this?
> > >
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
>  <mk...@griddynamics.com>
>

Re: paging vs streaming. spawn from (Processing a lot of results in Solr)

Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Roman,

Let me briefly explain  the design

special RequestParser stores servlet output stream into the context
https://github.com/m-khl/solr-patches/compare/streaming#L7R22

then special component injects special PostFilter/DelegatingCollector which
writes right into output
https://github.com/m-khl/solr-patches/compare/streaming#L2R146

here is how it streams the doc, you see it's lazy enough
https://github.com/m-khl/solr-patches/compare/streaming#L2R181

I mention that it disables later collectors
https://github.com/m-khl/solr-patches/compare/streaming#L2R57
hence, no facets with streaming, yet as well as memory consumption.

This test shows how it works
https://github.com/m-khl/solr-patches/compare/streaming#L15R115

all other code purposed for distributed search.



On Sat, Jul 27, 2013 at 4:44 PM, Roman Chyla <ro...@gmail.com> wrote:

> Mikhail,
> If your solution gives lazy loading of solr docs /and thus streaming of
> huge result lists/ it should be big YES!
> Roman
> On 27 Jul 2013 07:55, "Mikhail Khludnev" <mk...@griddynamics.com>
> wrote:
>
> > Otis,
> > You gave links to 'deep paging' when I asked about response streaming.
> > Let me understand. From my POV, deep paging is a special case for regular
> > search scenarios. We definitely need it in Solr. However, if we are
> talking
> > about data analytic like problems, when we need to select an "endless"
> > stream of responses (or store them in file as Roman did), 'deep paging'
> is
> > a suboptimal hack.
> > What's your vision on this?
> >
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
 <mk...@griddynamics.com>

Re: paging vs streaming. spawn from (Processing a lot of results in Solr)

Posted by Roman Chyla <ro...@gmail.com>.
Mikhail,
If your solution gives lazy loading of solr docs /and thus streaming of
huge result lists/ it should be big YES!
Roman
On 27 Jul 2013 07:55, "Mikhail Khludnev" <mk...@griddynamics.com> wrote:

> Otis,
> You gave links to 'deep paging' when I asked about response streaming.
> Let me understand. From my POV, deep paging is a special case for regular
> search scenarios. We definitely need it in Solr. However, if we are talking
> about data analytic like problems, when we need to select an "endless"
> stream of responses (or store them in file as Roman did), 'deep paging' is
> a suboptimal hack.
> What's your vision on this?
>