Should streaming place load on the app server?

[tedsolr <ts...@sciquest.com>]

The streaming API looks like it's meant to be run from the client app server
- very similar to a standard Solr search. When I run a basic streaming
operation the memory consumption occurs on the app server jvm, not the solr
server jvm. The opposite of what I was expecting. 

(pseudo code)
Stream A = new CloudSolrStream();
Stream B = new CloudSolrStream();
Stream C = new HashJoinStream(A, B);
Stream D = new SortStream(C);
Stream E = new ReducerStream(D);
E.open();

The SortStream is processed in memory when open() is called. Can the
processing be pushed off to the Solr cluster? Is that what the Parallel
stream will do - using worker collections?

confused,
Ted



--
View this message in context: http://lucene.472066.n3.nabble.com/Should-streaming-place-load-on-the-app-server-tp4288466.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Should streaming place load on the app server?

[tedsolr <ts...@sciquest.com>]

Since I'm using SolrJ as a conduit to Solr, to have the searches processed on
a Solr server I need to wrap everything in a ParallelStream object. Got it,
thanks!


Joel Bernstein wrote
> ....
> If you just use the Java API directly, the code executes in the VM where
> the code is run. You could use the ParallelStream to send the code to a
> SolrCloud worker to execute the code as well. In this scenario the code is
> serialized to a Streaming Expression and sent across the wire to the Solr
> node.
> 
> 
> Joel Bernstein
> http://joelsolr.blogspot.com/
> 
> On Fri, Jul 22, 2016 at 11:23 AM, tedsolr <

> tsmith@

> > wrote:
> 
>> The streaming API looks like it's meant to be run from the client app
>> server
>> - very similar to a standard Solr search. When I run a basic streaming
>> operation the memory consumption occurs on the app server jvm, not the
>> solr
>> server jvm. The opposite of what I was expecting.
>>
>> (pseudo code)
>> Stream A = new CloudSolrStream();
>> Stream B = new CloudSolrStream();
>> Stream C = new HashJoinStream(A, B);
>> Stream D = new SortStream(C);
>> Stream E = new ReducerStream(D);
>> E.open();
>>
>> The SortStream is processed in memory when open() is called. Can the
>> processing be pushed off to the Solr cluster? Is that what the Parallel
>> stream will do - using worker collections?
>>
>> confused,
>> Ted
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Should-streaming-place-load-on-the-app-server-tp4288466.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>





--
View this message in context: http://lucene.472066.n3.nabble.com/Should-streaming-place-load-on-the-app-server-tp4288466p4288521.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Should streaming place load on the app server?

[Joel Bernstein <jo...@gmail.com>]

Streaming Expression can be sent to any SolrCloud node in any collection.
You can setup collections that have no data and just execute the
expressions. The expressions reference other collections that hold data.

Collections that only execute expressions we can call "Worker Collections".
Collections that hold data we can call "Search Collections". This allows
you to have servers with a different specification for workers and search
nodes and offload the the workloads onto different collections.

The parallel function sends a streaming expression to N worker nodes. Each
worker node executes the expression and processes a partition of the data.
Both the search function and the topic function can be parallelized like
this. So operations like joins that wrap a search can be parallelized.

If you just use the Java API directly, the code executes in the VM where
the code is run. You could use the ParallelStream to send the code to a
SolrCloud worker to execute the code as well. In this scenario the code is
serialized to a Streaming Expression and sent across the wire to the Solr
node.






Joel Bernstein
http://joelsolr.blogspot.com/

On Fri, Jul 22, 2016 at 11:23 AM, tedsolr <ts...@sciquest.com> wrote:

> The streaming API looks like it's meant to be run from the client app
> server
> - very similar to a standard Solr search. When I run a basic streaming
> operation the memory consumption occurs on the app server jvm, not the solr
> server jvm. The opposite of what I was expecting.
>
> (pseudo code)
> Stream A = new CloudSolrStream();
> Stream B = new CloudSolrStream();
> Stream C = new HashJoinStream(A, B);
> Stream D = new SortStream(C);
> Stream E = new ReducerStream(D);
> E.open();
>
> The SortStream is processed in memory when open() is called. Can the
> processing be pushed off to the Solr cluster? Is that what the Parallel
> stream will do - using worker collections?
>
> confused,
> Ted
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Should-streaming-place-load-on-the-app-server-tp4288466.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>