You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Kamal Kishore Aggarwal <kk...@gmail.com> on 2018/10/02 12:21:17 UTC
Re: Solr Stream vs Export Request Handlers
Hi,
Thanks Jan & Joel.
Though I will evaluate the performance over my data, but based on your
experience, which one of the two is better in performance ?. Please suggest
Yeah, I know export does not get the data from all shards, but we can write
code to aggregate the data from all shards. But only if export is better
over stream.
Thanks
Kamal Kishore
On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein <jo...@gmail.com> wrote:
> The export handler does not do distributed search. So if you have a
> multi-shard collection you may have to use Streaming Expressions to get
> exports from all shards.
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
> > Hi,
> >
> > Yes, you can choose which to use, it should give you about same result.
> > If you already work with the Solr search API it would be the easiest for
> > you to consume /export as you don't need to learn the new syntax and
> parse
> > the Tuple response. However, if you need to do stuff with the docs as you
> > stream them from Solr, then streaming expressions lets you enrich the
> docs,
> > modify, join etc on the fly.
> >
> > PS: When the /export docs says it uses a streaming tecnique, it does NOT
> > mean that it has uses the solr feature streaming expressions :)
> >
> > --
> > Jan Høydahl, search solution architect
> > Cominvent AS - www.cominvent.com
> >
> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
> > kkroyal.192@gmail.com>:
> > >
> > > Hi,
> > >
> > > I have a requirement to fetch all data from a collection. One way is to
> > use
> > > streaming expression and other way is to use export.
> > >
> > > Streaming expression documentation says *streaming functions are
> designed
> > > to work with entire result sets rather then the top N results like
> normal
> > > search. This is supported by the /export handler.*
> > >
> > > Also, Export handler documentation says *this feature uses a stream
> > sorting
> > > technique that begins to send records within milliseconds and continues
> > to
> > > stream results until the entire result set has been sorted and
> exported.*
> > >
> > > These two statements concludes to me that for fetching entire results
> > > streaming expressions uses export handler and export handler uses
> stream,
> > > so, whether I use streaming expression or export handler, they are
> > > internally same and would have same performance. I am correct over here
> > to
> > > say so ?
> > >
> > >
> > > Ref Links:
> > >
> > > https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
> > > https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
> > >
> > >
> > > Regards
> > > Kamal Kishore
> >
> >
>
Re: Solr Stream vs Export Request Handlers
Posted by Joel Bernstein <jo...@gmail.com>.
I'm not sure why /stream is exporting faster then /export. It may be that
the different approaches in the client are the reason for the difference.
But the /export handler would be used in both scenarios if you specify
qt=/export in the search() Streaming Expression.
Joel Bernstein
http://joelsolr.blogspot.com/
On Thu, Oct 25, 2018 at 3:07 AM Kamal Kishore Aggarwal <
kkroyal.192@gmail.com> wrote:
> Any update on this.
>
> Regards
> Kamal
>
> On Thu, Oct 18, 2018 at 11:50 AM Kamal Kishore Aggarwal <
> kkroyal.192@gmail.com> wrote:
>
> > Hi,
> >
> > Thanks again Joel for your reply. I have noted your suggestions.
> >
> > I observed one more thing while using solrj to fetch the data using
> > /stream with export and direct /export. The solr QTime is almost same,
> > however elapsed time(total time) to fetch response in streaming with
> export
> > is better than direct /export (Streaming export taking 30% less than
> > /export).
> >
> > Is this also expected ?
> >
> > Regards
> > Kamal Kishore
> >
> >
> >
> > On Tue, Oct 16, 2018 at 3:21 AM Joel Bernstein <jo...@gmail.com>
> wrote:
> >
> >> Yes this is correct. But keep in mind Streaming Expression has a wide
> >> range
> >> of features that have nothing at all to do with the export handler. In
> >> general with Streaming Expressions you want to find the functions that
> get
> >> the job done using the least amount of work. The /export handler is
> often
> >> not the best choice. You'll want to read through the various streaming
> >> expressions to see if they might be more efficient for your use case.
> >>
> >>
> >> Joel Bernstein
> >> http://joelsolr.blogspot.com/
> >>
> >>
> >> On Mon, Oct 15, 2018 at 12:05 PM Kamal Kishore Aggarwal <
> >> kkroyal.192@gmail.com> wrote:
> >>
> >> > Hi,
> >> >
> >> > After I performed the test on my data, I found out that direct /export
> >> and
> >> > streaming expression with export, both are giving almost same response
> >> > time. This was also pointed out by *Jan Høydahl* in his reply.
> >> >
> >> > Also, the documentation says export feature uses stream sorting
> >> technique
> >> > and streaming expression also uses steam technique. So, are they
> >> internally
> >> > works in same fashion. Please confirm.
> >> >
> >> > Regards
> >> > Kamal Kishore
> >> >
> >> >
> >> >
> >> > On Tue, Oct 2, 2018 at 5:51 PM Kamal Kishore Aggarwal <
> >> > kkroyal.192@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > Thanks Jan & Joel.
> >> > >
> >> > > Though I will evaluate the performance over my data, but based on
> your
> >> > > experience, which one of the two is better in performance ?. Please
> >> > suggest
> >> > >
> >> > > Yeah, I know export does not get the data from all shards, but we
> can
> >> > > write code to aggregate the data from all shards. But only if export
> >> is
> >> > > better over stream.
> >> > >
> >> > > Thanks
> >> > > Kamal Kishore
> >> > >
> >> > >
> >> > > On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein <joelsolr@gmail.com
> >
> >> > > wrote:
> >> > >
> >> > >> The export handler does not do distributed search. So if you have a
> >> > >> multi-shard collection you may have to use Streaming Expressions to
> >> get
> >> > >> exports from all shards.
> >> > >>
> >> > >>
> >> > >> Joel Bernstein
> >> > >> http://joelsolr.blogspot.com/
> >> > >>
> >> > >>
> >> > >> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl <jan.asf@cominvent.com
> >
> >> > >> wrote:
> >> > >>
> >> > >> > Hi,
> >> > >> >
> >> > >> > Yes, you can choose which to use, it should give you about same
> >> > result.
> >> > >> > If you already work with the Solr search API it would be the
> >> easiest
> >> > for
> >> > >> > you to consume /export as you don't need to learn the new syntax
> >> and
> >> > >> parse
> >> > >> > the Tuple response. However, if you need to do stuff with the
> docs
> >> as
> >> > >> you
> >> > >> > stream them from Solr, then streaming expressions lets you enrich
> >> the
> >> > >> docs,
> >> > >> > modify, join etc on the fly.
> >> > >> >
> >> > >> > PS: When the /export docs says it uses a streaming tecnique, it
> >> does
> >> > NOT
> >> > >> > mean that it has uses the solr feature streaming expressions :)
> >> > >> >
> >> > >> > --
> >> > >> > Jan Høydahl, search solution architect
> >> > >> > Cominvent AS - www.cominvent.com
> >> > >> >
> >> > >> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
> >> > >> > kkroyal.192@gmail.com>:
> >> > >> > >
> >> > >> > > Hi,
> >> > >> > >
> >> > >> > > I have a requirement to fetch all data from a collection. One
> >> way is
> >> > >> to
> >> > >> > use
> >> > >> > > streaming expression and other way is to use export.
> >> > >> > >
> >> > >> > > Streaming expression documentation says *streaming functions
> are
> >> > >> designed
> >> > >> > > to work with entire result sets rather then the top N results
> >> like
> >> > >> normal
> >> > >> > > search. This is supported by the /export handler.*
> >> > >> > >
> >> > >> > > Also, Export handler documentation says *this feature uses a
> >> stream
> >> > >> > sorting
> >> > >> > > technique that begins to send records within milliseconds and
> >> > >> continues
> >> > >> > to
> >> > >> > > stream results until the entire result set has been sorted and
> >> > >> exported.*
> >> > >> > >
> >> > >> > > These two statements concludes to me that for fetching entire
> >> > results
> >> > >> > > streaming expressions uses export handler and export handler
> uses
> >> > >> stream,
> >> > >> > > so, whether I use streaming expression or export handler, they
> >> are
> >> > >> > > internally same and would have same performance. I am correct
> >> over
> >> > >> here
> >> > >> > to
> >> > >> > > say so ?
> >> > >> > >
> >> > >> > >
> >> > >> > > Ref Links:
> >> > >> > >
> >> > >> > >
> >> https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
> >> > >> > >
> >> https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
> >> > >> > >
> >> > >> > >
> >> > >> > > Regards
> >> > >> > > Kamal Kishore
> >> > >> >
> >> > >> >
> >> > >>
> >> > >
> >> >
> >>
> >
>
Re: Solr Stream vs Export Request Handlers
Posted by Kamal Kishore Aggarwal <kk...@gmail.com>.
Any update on this.
Regards
Kamal
On Thu, Oct 18, 2018 at 11:50 AM Kamal Kishore Aggarwal <
kkroyal.192@gmail.com> wrote:
> Hi,
>
> Thanks again Joel for your reply. I have noted your suggestions.
>
> I observed one more thing while using solrj to fetch the data using
> /stream with export and direct /export. The solr QTime is almost same,
> however elapsed time(total time) to fetch response in streaming with export
> is better than direct /export (Streaming export taking 30% less than
> /export).
>
> Is this also expected ?
>
> Regards
> Kamal Kishore
>
>
>
> On Tue, Oct 16, 2018 at 3:21 AM Joel Bernstein <jo...@gmail.com> wrote:
>
>> Yes this is correct. But keep in mind Streaming Expression has a wide
>> range
>> of features that have nothing at all to do with the export handler. In
>> general with Streaming Expressions you want to find the functions that get
>> the job done using the least amount of work. The /export handler is often
>> not the best choice. You'll want to read through the various streaming
>> expressions to see if they might be more efficient for your use case.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>> On Mon, Oct 15, 2018 at 12:05 PM Kamal Kishore Aggarwal <
>> kkroyal.192@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > After I performed the test on my data, I found out that direct /export
>> and
>> > streaming expression with export, both are giving almost same response
>> > time. This was also pointed out by *Jan Høydahl* in his reply.
>> >
>> > Also, the documentation says export feature uses stream sorting
>> technique
>> > and streaming expression also uses steam technique. So, are they
>> internally
>> > works in same fashion. Please confirm.
>> >
>> > Regards
>> > Kamal Kishore
>> >
>> >
>> >
>> > On Tue, Oct 2, 2018 at 5:51 PM Kamal Kishore Aggarwal <
>> > kkroyal.192@gmail.com>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > Thanks Jan & Joel.
>> > >
>> > > Though I will evaluate the performance over my data, but based on your
>> > > experience, which one of the two is better in performance ?. Please
>> > suggest
>> > >
>> > > Yeah, I know export does not get the data from all shards, but we can
>> > > write code to aggregate the data from all shards. But only if export
>> is
>> > > better over stream.
>> > >
>> > > Thanks
>> > > Kamal Kishore
>> > >
>> > >
>> > > On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein <jo...@gmail.com>
>> > > wrote:
>> > >
>> > >> The export handler does not do distributed search. So if you have a
>> > >> multi-shard collection you may have to use Streaming Expressions to
>> get
>> > >> exports from all shards.
>> > >>
>> > >>
>> > >> Joel Bernstein
>> > >> http://joelsolr.blogspot.com/
>> > >>
>> > >>
>> > >> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl <ja...@cominvent.com>
>> > >> wrote:
>> > >>
>> > >> > Hi,
>> > >> >
>> > >> > Yes, you can choose which to use, it should give you about same
>> > result.
>> > >> > If you already work with the Solr search API it would be the
>> easiest
>> > for
>> > >> > you to consume /export as you don't need to learn the new syntax
>> and
>> > >> parse
>> > >> > the Tuple response. However, if you need to do stuff with the docs
>> as
>> > >> you
>> > >> > stream them from Solr, then streaming expressions lets you enrich
>> the
>> > >> docs,
>> > >> > modify, join etc on the fly.
>> > >> >
>> > >> > PS: When the /export docs says it uses a streaming tecnique, it
>> does
>> > NOT
>> > >> > mean that it has uses the solr feature streaming expressions :)
>> > >> >
>> > >> > --
>> > >> > Jan Høydahl, search solution architect
>> > >> > Cominvent AS - www.cominvent.com
>> > >> >
>> > >> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
>> > >> > kkroyal.192@gmail.com>:
>> > >> > >
>> > >> > > Hi,
>> > >> > >
>> > >> > > I have a requirement to fetch all data from a collection. One
>> way is
>> > >> to
>> > >> > use
>> > >> > > streaming expression and other way is to use export.
>> > >> > >
>> > >> > > Streaming expression documentation says *streaming functions are
>> > >> designed
>> > >> > > to work with entire result sets rather then the top N results
>> like
>> > >> normal
>> > >> > > search. This is supported by the /export handler.*
>> > >> > >
>> > >> > > Also, Export handler documentation says *this feature uses a
>> stream
>> > >> > sorting
>> > >> > > technique that begins to send records within milliseconds and
>> > >> continues
>> > >> > to
>> > >> > > stream results until the entire result set has been sorted and
>> > >> exported.*
>> > >> > >
>> > >> > > These two statements concludes to me that for fetching entire
>> > results
>> > >> > > streaming expressions uses export handler and export handler uses
>> > >> stream,
>> > >> > > so, whether I use streaming expression or export handler, they
>> are
>> > >> > > internally same and would have same performance. I am correct
>> over
>> > >> here
>> > >> > to
>> > >> > > say so ?
>> > >> > >
>> > >> > >
>> > >> > > Ref Links:
>> > >> > >
>> > >> > >
>> https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
>> > >> > >
>> https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
>> > >> > >
>> > >> > >
>> > >> > > Regards
>> > >> > > Kamal Kishore
>> > >> >
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: Solr Stream vs Export Request Handlers
Posted by Kamal Kishore Aggarwal <kk...@gmail.com>.
Hi,
Thanks again Joel for your reply. I have noted your suggestions.
I observed one more thing while using solrj to fetch the data using /stream
with export and direct /export. The solr QTime is almost same, however
elapsed time(total time) to fetch response in streaming with export is
better than direct /export (Streaming export taking 30% less than /export).
Is this also expected ?
Regards
Kamal Kishore
On Tue, Oct 16, 2018 at 3:21 AM Joel Bernstein <jo...@gmail.com> wrote:
> Yes this is correct. But keep in mind Streaming Expression has a wide range
> of features that have nothing at all to do with the export handler. In
> general with Streaming Expressions you want to find the functions that get
> the job done using the least amount of work. The /export handler is often
> not the best choice. You'll want to read through the various streaming
> expressions to see if they might be more efficient for your use case.
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Mon, Oct 15, 2018 at 12:05 PM Kamal Kishore Aggarwal <
> kkroyal.192@gmail.com> wrote:
>
> > Hi,
> >
> > After I performed the test on my data, I found out that direct /export
> and
> > streaming expression with export, both are giving almost same response
> > time. This was also pointed out by *Jan Høydahl* in his reply.
> >
> > Also, the documentation says export feature uses stream sorting technique
> > and streaming expression also uses steam technique. So, are they
> internally
> > works in same fashion. Please confirm.
> >
> > Regards
> > Kamal Kishore
> >
> >
> >
> > On Tue, Oct 2, 2018 at 5:51 PM Kamal Kishore Aggarwal <
> > kkroyal.192@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Thanks Jan & Joel.
> > >
> > > Though I will evaluate the performance over my data, but based on your
> > > experience, which one of the two is better in performance ?. Please
> > suggest
> > >
> > > Yeah, I know export does not get the data from all shards, but we can
> > > write code to aggregate the data from all shards. But only if export is
> > > better over stream.
> > >
> > > Thanks
> > > Kamal Kishore
> > >
> > >
> > > On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein <jo...@gmail.com>
> > > wrote:
> > >
> > >> The export handler does not do distributed search. So if you have a
> > >> multi-shard collection you may have to use Streaming Expressions to
> get
> > >> exports from all shards.
> > >>
> > >>
> > >> Joel Bernstein
> > >> http://joelsolr.blogspot.com/
> > >>
> > >>
> > >> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl <ja...@cominvent.com>
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > Yes, you can choose which to use, it should give you about same
> > result.
> > >> > If you already work with the Solr search API it would be the easiest
> > for
> > >> > you to consume /export as you don't need to learn the new syntax and
> > >> parse
> > >> > the Tuple response. However, if you need to do stuff with the docs
> as
> > >> you
> > >> > stream them from Solr, then streaming expressions lets you enrich
> the
> > >> docs,
> > >> > modify, join etc on the fly.
> > >> >
> > >> > PS: When the /export docs says it uses a streaming tecnique, it does
> > NOT
> > >> > mean that it has uses the solr feature streaming expressions :)
> > >> >
> > >> > --
> > >> > Jan Høydahl, search solution architect
> > >> > Cominvent AS - www.cominvent.com
> > >> >
> > >> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
> > >> > kkroyal.192@gmail.com>:
> > >> > >
> > >> > > Hi,
> > >> > >
> > >> > > I have a requirement to fetch all data from a collection. One way
> is
> > >> to
> > >> > use
> > >> > > streaming expression and other way is to use export.
> > >> > >
> > >> > > Streaming expression documentation says *streaming functions are
> > >> designed
> > >> > > to work with entire result sets rather then the top N results like
> > >> normal
> > >> > > search. This is supported by the /export handler.*
> > >> > >
> > >> > > Also, Export handler documentation says *this feature uses a
> stream
> > >> > sorting
> > >> > > technique that begins to send records within milliseconds and
> > >> continues
> > >> > to
> > >> > > stream results until the entire result set has been sorted and
> > >> exported.*
> > >> > >
> > >> > > These two statements concludes to me that for fetching entire
> > results
> > >> > > streaming expressions uses export handler and export handler uses
> > >> stream,
> > >> > > so, whether I use streaming expression or export handler, they are
> > >> > > internally same and would have same performance. I am correct over
> > >> here
> > >> > to
> > >> > > say so ?
> > >> > >
> > >> > >
> > >> > > Ref Links:
> > >> > >
> > >> > >
> https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
> > >> > >
> https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
> > >> > >
> > >> > >
> > >> > > Regards
> > >> > > Kamal Kishore
> > >> >
> > >> >
> > >>
> > >
> >
>
Re: Solr Stream vs Export Request Handlers
Posted by Joel Bernstein <jo...@gmail.com>.
Yes this is correct. But keep in mind Streaming Expression has a wide range
of features that have nothing at all to do with the export handler. In
general with Streaming Expressions you want to find the functions that get
the job done using the least amount of work. The /export handler is often
not the best choice. You'll want to read through the various streaming
expressions to see if they might be more efficient for your use case.
Joel Bernstein
http://joelsolr.blogspot.com/
On Mon, Oct 15, 2018 at 12:05 PM Kamal Kishore Aggarwal <
kkroyal.192@gmail.com> wrote:
> Hi,
>
> After I performed the test on my data, I found out that direct /export and
> streaming expression with export, both are giving almost same response
> time. This was also pointed out by *Jan Høydahl* in his reply.
>
> Also, the documentation says export feature uses stream sorting technique
> and streaming expression also uses steam technique. So, are they internally
> works in same fashion. Please confirm.
>
> Regards
> Kamal Kishore
>
>
>
> On Tue, Oct 2, 2018 at 5:51 PM Kamal Kishore Aggarwal <
> kkroyal.192@gmail.com>
> wrote:
>
> > Hi,
> >
> > Thanks Jan & Joel.
> >
> > Though I will evaluate the performance over my data, but based on your
> > experience, which one of the two is better in performance ?. Please
> suggest
> >
> > Yeah, I know export does not get the data from all shards, but we can
> > write code to aggregate the data from all shards. But only if export is
> > better over stream.
> >
> > Thanks
> > Kamal Kishore
> >
> >
> > On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein <jo...@gmail.com>
> > wrote:
> >
> >> The export handler does not do distributed search. So if you have a
> >> multi-shard collection you may have to use Streaming Expressions to get
> >> exports from all shards.
> >>
> >>
> >> Joel Bernstein
> >> http://joelsolr.blogspot.com/
> >>
> >>
> >> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl <ja...@cominvent.com>
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > Yes, you can choose which to use, it should give you about same
> result.
> >> > If you already work with the Solr search API it would be the easiest
> for
> >> > you to consume /export as you don't need to learn the new syntax and
> >> parse
> >> > the Tuple response. However, if you need to do stuff with the docs as
> >> you
> >> > stream them from Solr, then streaming expressions lets you enrich the
> >> docs,
> >> > modify, join etc on the fly.
> >> >
> >> > PS: When the /export docs says it uses a streaming tecnique, it does
> NOT
> >> > mean that it has uses the solr feature streaming expressions :)
> >> >
> >> > --
> >> > Jan Høydahl, search solution architect
> >> > Cominvent AS - www.cominvent.com
> >> >
> >> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
> >> > kkroyal.192@gmail.com>:
> >> > >
> >> > > Hi,
> >> > >
> >> > > I have a requirement to fetch all data from a collection. One way is
> >> to
> >> > use
> >> > > streaming expression and other way is to use export.
> >> > >
> >> > > Streaming expression documentation says *streaming functions are
> >> designed
> >> > > to work with entire result sets rather then the top N results like
> >> normal
> >> > > search. This is supported by the /export handler.*
> >> > >
> >> > > Also, Export handler documentation says *this feature uses a stream
> >> > sorting
> >> > > technique that begins to send records within milliseconds and
> >> continues
> >> > to
> >> > > stream results until the entire result set has been sorted and
> >> exported.*
> >> > >
> >> > > These two statements concludes to me that for fetching entire
> results
> >> > > streaming expressions uses export handler and export handler uses
> >> stream,
> >> > > so, whether I use streaming expression or export handler, they are
> >> > > internally same and would have same performance. I am correct over
> >> here
> >> > to
> >> > > say so ?
> >> > >
> >> > >
> >> > > Ref Links:
> >> > >
> >> > > https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
> >> > > https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
> >> > >
> >> > >
> >> > > Regards
> >> > > Kamal Kishore
> >> >
> >> >
> >>
> >
>
Re: Solr Stream vs Export Request Handlers
Posted by Kamal Kishore Aggarwal <kk...@gmail.com>.
Hi,
After I performed the test on my data, I found out that direct /export and
streaming expression with export, both are giving almost same response
time. This was also pointed out by *Jan Høydahl* in his reply.
Also, the documentation says export feature uses stream sorting technique
and streaming expression also uses steam technique. So, are they internally
works in same fashion. Please confirm.
Regards
Kamal Kishore
On Tue, Oct 2, 2018 at 5:51 PM Kamal Kishore Aggarwal <kk...@gmail.com>
wrote:
> Hi,
>
> Thanks Jan & Joel.
>
> Though I will evaluate the performance over my data, but based on your
> experience, which one of the two is better in performance ?. Please suggest
>
> Yeah, I know export does not get the data from all shards, but we can
> write code to aggregate the data from all shards. But only if export is
> better over stream.
>
> Thanks
> Kamal Kishore
>
>
> On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein <jo...@gmail.com>
> wrote:
>
>> The export handler does not do distributed search. So if you have a
>> multi-shard collection you may have to use Streaming Expressions to get
>> exports from all shards.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl <ja...@cominvent.com>
>> wrote:
>>
>> > Hi,
>> >
>> > Yes, you can choose which to use, it should give you about same result.
>> > If you already work with the Solr search API it would be the easiest for
>> > you to consume /export as you don't need to learn the new syntax and
>> parse
>> > the Tuple response. However, if you need to do stuff with the docs as
>> you
>> > stream them from Solr, then streaming expressions lets you enrich the
>> docs,
>> > modify, join etc on the fly.
>> >
>> > PS: When the /export docs says it uses a streaming tecnique, it does NOT
>> > mean that it has uses the solr feature streaming expressions :)
>> >
>> > --
>> > Jan Høydahl, search solution architect
>> > Cominvent AS - www.cominvent.com
>> >
>> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
>> > kkroyal.192@gmail.com>:
>> > >
>> > > Hi,
>> > >
>> > > I have a requirement to fetch all data from a collection. One way is
>> to
>> > use
>> > > streaming expression and other way is to use export.
>> > >
>> > > Streaming expression documentation says *streaming functions are
>> designed
>> > > to work with entire result sets rather then the top N results like
>> normal
>> > > search. This is supported by the /export handler.*
>> > >
>> > > Also, Export handler documentation says *this feature uses a stream
>> > sorting
>> > > technique that begins to send records within milliseconds and
>> continues
>> > to
>> > > stream results until the entire result set has been sorted and
>> exported.*
>> > >
>> > > These two statements concludes to me that for fetching entire results
>> > > streaming expressions uses export handler and export handler uses
>> stream,
>> > > so, whether I use streaming expression or export handler, they are
>> > > internally same and would have same performance. I am correct over
>> here
>> > to
>> > > say so ?
>> > >
>> > >
>> > > Ref Links:
>> > >
>> > > https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
>> > > https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
>> > >
>> > >
>> > > Regards
>> > > Kamal Kishore
>> >
>> >
>>
>