You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Lee Carroll <le...@googlemail.com> on 2018/04/16 15:56:39 UTC
custom response writer which extends RawResponseWriter fails when
shards > 1
I've created a custom response writer which extends RawResponseWriter. The
basic operation is to output a single field value from the top matching doc
as the entire response. This works when shards = 1 but fails when shards
are greater than 1.
I throw an error if the field in question is missing from the top doc. This
happens when individual shards are being searched and only id and score are
returned. I'm sure I've committed a basic error.
Lee C
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Mikhail Khludnev <mk...@apache.org>.
You can introduce own searchHandler with wt invariant.
When aggreator request slaves it will use regular /select with default
wt=javabin (it's condrolled by shards.qt, btw)
Providing such comprehensive application logic on top of neat
solrconfig.xml is not the best idea, though.
On Thu, Apr 19, 2018 at 9:57 PM, Lee Carroll <le...@googlemail.com>
wrote:
> Default works. However in the case (which is maybe my case) where request
> handler implies a response type and really should be locked down to prevent
> abuse or error you could argue invariant is needed.
> I guess its also not very elegant having an arbitrary rule, no wt as
> invariant in cloud mode etc.
>
>
> lee c
>
> On 19 April 2018 at 19:00, Mikhail Khludnev <mk...@apache.org> wrote:
>
> > what if you put it into "defaults"?
> >
> > On Thu, Apr 19, 2018 at 8:42 PM, Lee Carroll <
> lee.a.carroll@googlemail.com
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I rewrote all of my tests to use SolrCloudTestCase rather than
> > > SolrTestCaseJ4
> > > and was able to replicate the responsewriter issue and debug with a
> > sharded
> > > collection. It turned out the issue was not with my response writer
> > really
> > > but rather my config.
> > >
> > > <requestHandler name="/content" class="solr.SearchHandler">
> > > ....
> > >
> > > <lst name="invariants">
> > > <str name="wt">content</str>
> > > </lst>
> > >
> > > </requestHandler>
> > >
> > > In cloud mode having wt as an invariant breaks the collation of results
> > > from shards. Now I'm sure this is a common mistake which I've repeated
> > > (blush) but I do sort of want to actually implement my request handler
> in
> > > this way. Is their a way to have a request handler support a single
> > > response writer but still work in cloud mode ?
> > >
> > > Could this be considered a bug ?
> > >
> > > Lee C
> > >
> > > On 18 April 2018 at 13:13, Mikhail Khludnev <mk...@apache.org> wrote:
> > >
> > > > Injecting headers might require deeper customisation up to
> establishing
> > > own
> > > > filter or so.
> > > > Speaking regarding your own WT, there might be some issues because
> > > usually
> > > > it's not a big deal to use one wt for responding user query like
> > (wt=csv)
> > > > and wt=javabin in internal communication between aggregator and
> slaves
> > > like
> > > > it happens in wt=csv query.
> > > >
> > > > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll <
> > > lee.a.carroll@googlemail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Inventive. I need to control content-type of the response from the
> > > > document
> > > > > field value. I have the actual content field and the content-type
> > field
> > > > to
> > > > > use configured in the response writer. I've just noticed that the
> > xslt
> > > > > transformer allows you to do this but not controlled by document
> > > values.
> > > > I
> > > > > may also need to set some headers based on content-type and perhaps
> > > > content
> > > > > size, accept ranges comes to mind. Although I might be getting
> ahead
> > of
> > > > > myself.
> > > > >
> > > > >
> > > > >
> > > > > On 18 April 2018 at 12:05, Mikhail Khludnev <mk...@apache.org>
> wrote:
> > > > >
> > > > > > well ..
> > > > > > what if
> > > > > > http://localhost:8983/solr/images/select?fl=content&q=id:
> > > > > 1&start=1&wt=csv&
> > > > > > csv.separator=&csv.encapsulator&csv.null=null
> > > > > > ?
> > > > > >
> > > > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll <
> > > > > lee.a.carroll@googlemail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > sorry cut n paste error i'd get
> > > > > > >
> > > > > > > {
> > > > > > > "responseHeader":{
> > > > > > > "zkConnected":true,
> > > > > > > "status":0,
> > > > > > > "QTime":0,
> > > > > > > "params":{
> > > > > > > "q":"*:*",
> > > > > > > "fl":"content",
> > > > > > > "rows":"1"}},
> > > > > > > "response":{"numFound":1,"start":0,"docs":[
> > > > > > > {
> > > > > > > "content":"my-content-value"}]
> > > > > > > }}
> > > > > > >
> > > > > > >
> > > > > > > but you get my point
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 18 April 2018 at 11:13, Lee Carroll <
> > > lee.a.carroll@googlemail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > for http://localhost:8983/solr/images/select?fl=content&q=id
> :
> > > > > 1&start=1
> > > > > > > >
> > > > > > > > I'd get
> > > > > > > >
> > > > > > > > {
> > > > > > > > "responseHeader":{
> > > > > > > > "zkConnected":true,
> > > > > > > > "status":0,
> > > > > > > > "QTime":1,
> > > > > > > > "params":{
> > > > > > > > "q":"*:*",
> > > > > > > > "_":"1524046333220"}},
> > > > > > > > "response":{"numFound":1,"start":0,"docs":[
> > > > > > > > {
> > > > > > > > "id":"1",
> > > > > > > > "content":"my-content-value",
> > > > > > > > "*content-type*":"text/plain"}]
> > > > > > > > }}
> > > > > > > >
> > > > > > > > when i want
> > > > > > > >
> > > > > > > > my-content-value
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev <mkhl@apache.org
> >
> > > > wrote:
> > > > > > > >
> > > > > > > >> Lee, from this description I don see why it can't be
> addressed
> > > by
> > > > > > > fl,rows
> > > > > > > >> params. What makes it different form the typical Solr usage?
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll <
> > > > > > > >> lee.a.carroll@googlemail.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Sure, we want to return a single field's value for the top
> > > > > matching
> > > > > > > >> > document for a given query. Bare content rather than a
> full
> > > > search
> > > > > > > >> result
> > > > > > > >> > listing.
> > > > > > > >> >
> > > > > > > >> > To be concrete:
> > > > > > > >> >
> > > > > > > >> > For a schema of fields id [unique key],
> > > > > > content[stored],content-type[
> > > > > > > >> > stored]
> > > > > > > >> > For a request:
> > > > > > > >> >
> > > > > > > >> > 1. Request URL:
> > > > > > > >> > https://localhost/solr/content?q=id:1
> > > > > > > >> > 2. Request Method:
> > > > > > > >> > GET
> > > > > > > >> >
> > > > > > > >> > We get a response
> > > > > > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type:
> > > > [content-type
> > > > > > > value]
> > > > > > > >> >
> > > > > > > >> > and the body to be the raw value of content
> > > > > > > >> >
> > > > > > > >> > In short clients consume directly the most relevant
> > "content"
> > > > > > returned
> > > > > > > >> from
> > > > > > > >> > solr queries they construct.
> > > > > > > >> >
> > > > > > > >> > Naively I've implemented a subclass of RawResponseWriter
> > which
> > > > > takes
> > > > > > > the
> > > > > > > >> > first docs values and adds them to the appended "content"
> > > > stream.
> > > > > > > >> Should I
> > > > > > > >> > selectively add the content stream depending on if this is
> > the
> > > > > final
> > > > > > > >> > aggregation of cloud results (and provide a base class
> > writer
> > > to
> > > > > act
> > > > > > > if
> > > > > > > >> > not), if so how do I know its the final aggregation. Or is
> > > > adding
> > > > > > the
> > > > > > > >> > content stream within the response writer a bad idea.
> Should
> > > > that
> > > > > be
> > > > > > > >> being
> > > > > > > >> > added to the response somewhere else?
> > > > > > > >> >
> > > > > > > >> > Failing all of the above is asking about response writer
> an
> > X
> > > /
> > > > Y
> > > > > > > >> problem.
> > > > > > > >> > Is their a better way to achieve the above. I'd looked at
> > > > > > transforming
> > > > > > > >> > response xml but that seemed not to offer a complete bare
> > > slate.
> > > > > > > >> >
> > > > > > > >> > Cheers Lee C
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On 17 April 2018 at 21:36, Mikhail Khludnev <
> > mkhl@apache.org>
> > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > In distributed search response writer is used twice
> > > > > > > >> > > https://lucene.apache.org/solr/guide/7_1/distributed-
> > > > > > requests.html
> > > > > > > >> > > once slave node that's where response writer yields
> "json"
> > > > > content
> > > > > > > >> and it
> > > > > > > >> > > upset aggregator node which is expect only javabin.
> > > > > > > >> > > I hardly can comment on rrw, it's probably used for
> > > responding
> > > > > > > >> separate
> > > > > > > >> > > files in distrib=false mode.
> > > > > > > >> > > You can start from describing why you need to create own
> > > > > response
> > > > > > > >> writer.
> > > > > > > >> > >
> > > > > > > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <
> > > > > > > >> > lee.a.carroll@googlemail.com
> > > > > > > >> > > >
> > > > > > > >> > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Ok. My expectation was the response writer would not
> be
> > > used
> > > > > > until
> > > > > > > >> the
> > > > > > > >> > > > final serialization of the result. If my response
> writer
> > > > > breaks
> > > > > > > the
> > > > > > > >> > > > response writer contract, exactly the way
> > > rawResponseWriter
> > > > > does
> > > > > > > and
> > > > > > > >> > just
> > > > > > > >> > > > out puts a filed value how does that work? Does
> > > > > > rawResponseWriter
> > > > > > > >> > support
> > > > > > > >> > > > cloud mode?
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev <
> > > > mkhl@apache.org>
> > > > > > > >> wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > That's what should happen.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Expected mime type application/octet-stream but got
> > > > > > > >> application/json.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Distributed search coordinator expect to merge slave
> > > > > responses
> > > > > > > in
> > > > > > > >> > > javabin
> > > > > > > >> > > > > format. But slave's wt indicated json.
> > > > > > > >> > > > > As far as I know only javabin might be used to
> > > distributed
> > > > > > > search
> > > > > > > >> > > > > underneath. Coordinator itself might yield json.
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
> > > > > > > >> > > > lee.a.carroll@googlemail.com
> > > > > > > >> > > > > >
> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > Sure
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > with 1 shard 1 replica this request works fine
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > 1. Request URL:
> > > > > > > >> > > > > > http://localhost:8983/solr/images/image?q=id:1
> > > > > > > >> > > > > > 2. Request Method:
> > > > > > > >> > > > > > GET
> > > > > > > >> > > > > > 3. Status Code:
> > > > > > > >> > > > > > 200 OK
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > logs are clean
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > with 2 shards 2 replicas the same request fails
> and
> > in
> > > > the
> > > > > > > logs
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images
> s:shard2
> > > > > > > r:core_node7
> > > > > > > >> > > > > > x:images_shard2_replica_n4]
> > > > org.apache.solr.core.SolrCore;
> > > > > > > >> > > > > > [images_shard2_replica_n4] webapp=/solr
> path=/image
> > > > > > > >> > > > > > params={df=text&distrib=false&
> > > qt=/image&fl=id&fl=score&
> > > > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > > > > > >> > > > > > http://10.224.30.207:8983/
> > > > solr/images_shard2_replica_n4/
> > > > > > > >> > > > > > |http://10.224.30.207:7574/
> > > > solr/images_shard2_replica_n6/
> > > > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
> > > > > > 1523971232039&isShard=true&wt=
> > > > > > > >> > javabin}
> > > > > > > >> > > > > > hits=0 status=0 QTime=0
> > > > > > > >> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images
> s:shard1
> > > > > > > r:core_node3
> > > > > > > >> > > > > > x:images_shard1_replica_n1]
> > > > org.apache.solr.common.SolrExc
> > > > > > > >> eption;
> > > > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
> > > > > > > >> > > RemoteSolrException:
> > > > > > > >> > > > > > Error
> > > > > > > >> > > > > > from server at http://10.224.30.207:8983/
> > > > > > > >> > > solr/images_shard2_replica_n4
> > > > > > > >> > > > :
> > > > > > > >> > > > > > Expected mime type application/octet-stream but
> got
> > > > > > > >> > application/json.
> > > > > > > >> > > > > > at
> > > > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
> > > > > > > >> > > > > > executeMethod(HttpSolrClient.java:607)
> > > > > > > >> > > > > > at
> > > > > > > >> > > > > > org.apache.solr.client.solrj.
> > > > impl.HttpSolrClient.request(
> > > > > > > >> > > > > > HttpSolrClient.java:255)
> > > > > > > >> > > > > > at
> > > > > > > >> > > > > > org.apache.solr.client.solrj.
> > > > impl.HttpSolrClient.request(
> > > > > > > >> > > > > > HttpSolrClient.java:244)
> > > > > > > >> > > > > > at
> > > > > > > >> > > > > > org.apache.solr.client.solrj.
> impl.LBHttpSolrClient.
> > > > > > > >> > > > > > doRequest(LBHttpSolrClient.java:483)
> > > > > > > >> > > > > > at
> > > > > > > >> > > > > > org.apache.solr.client.solrj.
> > > > > impl.LBHttpSolrClient.request(
> > > > > > > >> > > > > > LBHttpSolrClient.java:413)
> > > > > > > >> > > > > > at
> > > > > > > >> > > > > > org.apache.solr.handler.component.
> > > > > HttpShardHandlerFactory.
> > > > > > > >> > > > > > makeLoadBalancedRequest(
> > HttpShardHandlerFactory.java:
> > > > 273)
> > > > > > > >> > > > > > at
> > > > > > > >> > > > > > org.apache.solr.handler.
> component.HttpShardHandler.
> > > > > > > >> > lambda$submit$0(
> > > > > > > >> > > > > > HttpShardHandler.java:175)
> > > > > > > >> > > > > > at java.util.concurrent.
> FutureTask.run(FutureTask.
> > > > > java:266)
> > > > > > > >> > > > > > at java.util.concurrent.
> Executors$RunnableAdapter.
> > > > > > > >> > > > > call(Executors.java:511)
> > > > > > > >> > > > > > at java.util.concurrent.
> FutureTask.run(FutureTask.
> > > > > java:266)
> > > > > > > >> > > > > > at
> > > > > > > >> > > > > > com.codahale.metrics.InstrumentedExecutorService$
> > > > > > > >> > > > > InstrumentedRunnable.run(
> > > > > > > >> > > > > > InstrumentedExecutorService.java:176)
> > > > > > > >> > > > > > at
> > > > > > > >> > > > > > org.apache.solr.common.util.ExecutorUtil$
> > > > > > > >> > MDCAwareThreadPoolExecutor.
> > > > > > > >> > > > > > lambda$execute$0(ExecutorUtil.java:188)
> > > > > > > >> > > > > > at
> > > > > > > >> > > > > > java.util.concurrent.
> ThreadPoolExecutor.runWorker(
> > > > > > > >> > > > > > ThreadPoolExecutor.java:1142)
> > > > > > > >> > > > > > at
> > > > > > > >> > > > > > java.util.concurrent.
> ThreadPoolExecutor$Worker.run(
> > > > > > > >> > > > > > ThreadPoolExecutor.java:617)
> > > > > > > >> > > > > > at java.lang.Thread.run(Thread.java:745)
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images
> s:shard1
> > > > > > > r:core_node3
> > > > > > > >> > > > > > x:images_shard1_replica_n1]
> > > > org.apache.solr.core.SolrCore;
> > > > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr
> path=/image
> > > > > > > >> > params={q=id:1}
> > > > > > > >> > > > > > status=200 QTime=17
> > > > > > > >> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images
> s:shard1
> > > > > > > r:core_node3
> > > > > > > >> > > > > > x:images_shard1_replica_n1]
> > > > org.apache.solr.core.SolrCore;
> > > > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr
> path=/image
> > > > > > > >> > > > > > params={df=text&distrib=false&
> > > qt=/image&fl=id&fl=score&
> > > > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > > > > > >> > > > > > http://10.224.30.207:8983/
> > > > solr/images_shard1_replica_n1/
> > > > > > > >> > > > > > |http://10.224.30.207:7574/
> > > > solr/images_shard1_replica_n2/
> > > > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
> > > > > > 1523971232039&isShard=true&wt=
> > > > > > > >> > javabin}
> > > > > > > >> > > > > > hits=1 status=0 QTime=2
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > I've implemented getcontenttype simply as
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > @Override
> > > > > > > >> > > > > > public String getContentType(SolrQueryRequest
> > > request,
> > > > > > > >> > > > > > SolrQueryResponse response) {
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > return "application/json;charset=utf-8";
> > > > > > > >> > > > > > }
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev <
> > > > > > mkhl@apache.org>
> > > > > > > >> > wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > Lee,
> > > > > > > >> > > > > > > It's worth to send a stacktrace for such kind of
> > > > > > inquiries.
> > > > > > > >> > > > > > > I guess it goes from QueryComponent.mergeIds()
> or
> > > so.
> > > > > > Shard
> > > > > > > >> > > response
> > > > > > > >> > > > > > should
> > > > > > > >> > > > > > > contains <uniqueKey> from schema.xml field.
> > > > > > > >> > > > > > > I encounter something like this while
> > > troubleshooting
> > > > > > > >> > > > > > > https://lucene.apache.org/
> > > > solr/guide/6_6/transforming-
> > > > > > > >> > > > > > > result-documents.html#
> > TransformingResultDocuments-
> > > > > > > >> > > > > > > CoresandCollectionsinSolrCloud
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
> > > > > > > >> > > > > > lee.a.carroll@googlemail.com
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > I've created a custom response writer which
> > > extends
> > > > > > > >> > > > > RawResponseWriter.
> > > > > > > >> > > > > > > The
> > > > > > > >> > > > > > > > basic operation is to output a single field
> > value
> > > > from
> > > > > > the
> > > > > > > >> top
> > > > > > > >> > > > > matching
> > > > > > > >> > > > > > > doc
> > > > > > > >> > > > > > > > as the entire response. This works when shards
> > = 1
> > > > but
> > > > > > > fails
> > > > > > > >> > when
> > > > > > > >> > > > > > shards
> > > > > > > >> > > > > > > > are greater than 1.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > I throw an error if the field in question is
> > > missing
> > > > > > from
> > > > > > > >> the
> > > > > > > >> > top
> > > > > > > >> > > > > doc.
> > > > > > > >> > > > > > > This
> > > > > > > >> > > > > > > > happens when individual shards are being
> > searched
> > > > and
> > > > > > only
> > > > > > > >> id
> > > > > > > >> > and
> > > > > > > >> > > > > score
> > > > > > > >> > > > > > > are
> > > > > > > >> > > > > > > > returned. I'm sure I've committed a basic
> error.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Lee C
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > --
> > > > > > > >> > > > > > > Sincerely yours
> > > > > > > >> > > > > > > Mikhail Khludnev
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > --
> > > > > > > >> > > > > Sincerely yours
> > > > > > > >> > > > > Mikhail Khludnev
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > --
> > > > > > > >> > > Sincerely yours
> > > > > > > >> > > Mikhail Khludnev
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Sincerely yours
> > > > > > > >> Mikhail Khludnev
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Sincerely yours
> > > > > > Mikhail Khludnev
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sincerely yours
> > > > Mikhail Khludnev
> > > >
> > >
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> >
>
--
Sincerely yours
Mikhail Khludnev
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Lee Carroll <le...@googlemail.com>.
Default works. However in the case (which is maybe my case) where request
handler implies a response type and really should be locked down to prevent
abuse or error you could argue invariant is needed.
I guess its also not very elegant having an arbitrary rule, no wt as
invariant in cloud mode etc.
lee c
On 19 April 2018 at 19:00, Mikhail Khludnev <mk...@apache.org> wrote:
> what if you put it into "defaults"?
>
> On Thu, Apr 19, 2018 at 8:42 PM, Lee Carroll <lee.a.carroll@googlemail.com
> >
> wrote:
>
> > Hi,
> >
> > I rewrote all of my tests to use SolrCloudTestCase rather than
> > SolrTestCaseJ4
> > and was able to replicate the responsewriter issue and debug with a
> sharded
> > collection. It turned out the issue was not with my response writer
> really
> > but rather my config.
> >
> > <requestHandler name="/content" class="solr.SearchHandler">
> > ....
> >
> > <lst name="invariants">
> > <str name="wt">content</str>
> > </lst>
> >
> > </requestHandler>
> >
> > In cloud mode having wt as an invariant breaks the collation of results
> > from shards. Now I'm sure this is a common mistake which I've repeated
> > (blush) but I do sort of want to actually implement my request handler in
> > this way. Is their a way to have a request handler support a single
> > response writer but still work in cloud mode ?
> >
> > Could this be considered a bug ?
> >
> > Lee C
> >
> > On 18 April 2018 at 13:13, Mikhail Khludnev <mk...@apache.org> wrote:
> >
> > > Injecting headers might require deeper customisation up to establishing
> > own
> > > filter or so.
> > > Speaking regarding your own WT, there might be some issues because
> > usually
> > > it's not a big deal to use one wt for responding user query like
> (wt=csv)
> > > and wt=javabin in internal communication between aggregator and slaves
> > like
> > > it happens in wt=csv query.
> > >
> > > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll <
> > lee.a.carroll@googlemail.com
> > > >
> > > wrote:
> > >
> > > > Inventive. I need to control content-type of the response from the
> > > document
> > > > field value. I have the actual content field and the content-type
> field
> > > to
> > > > use configured in the response writer. I've just noticed that the
> xslt
> > > > transformer allows you to do this but not controlled by document
> > values.
> > > I
> > > > may also need to set some headers based on content-type and perhaps
> > > content
> > > > size, accept ranges comes to mind. Although I might be getting ahead
> of
> > > > myself.
> > > >
> > > >
> > > >
> > > > On 18 April 2018 at 12:05, Mikhail Khludnev <mk...@apache.org> wrote:
> > > >
> > > > > well ..
> > > > > what if
> > > > > http://localhost:8983/solr/images/select?fl=content&q=id:
> > > > 1&start=1&wt=csv&
> > > > > csv.separator=&csv.encapsulator&csv.null=null
> > > > > ?
> > > > >
> > > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll <
> > > > lee.a.carroll@googlemail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > sorry cut n paste error i'd get
> > > > > >
> > > > > > {
> > > > > > "responseHeader":{
> > > > > > "zkConnected":true,
> > > > > > "status":0,
> > > > > > "QTime":0,
> > > > > > "params":{
> > > > > > "q":"*:*",
> > > > > > "fl":"content",
> > > > > > "rows":"1"}},
> > > > > > "response":{"numFound":1,"start":0,"docs":[
> > > > > > {
> > > > > > "content":"my-content-value"}]
> > > > > > }}
> > > > > >
> > > > > >
> > > > > > but you get my point
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 18 April 2018 at 11:13, Lee Carroll <
> > lee.a.carroll@googlemail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > for http://localhost:8983/solr/images/select?fl=content&q=id:
> > > > 1&start=1
> > > > > > >
> > > > > > > I'd get
> > > > > > >
> > > > > > > {
> > > > > > > "responseHeader":{
> > > > > > > "zkConnected":true,
> > > > > > > "status":0,
> > > > > > > "QTime":1,
> > > > > > > "params":{
> > > > > > > "q":"*:*",
> > > > > > > "_":"1524046333220"}},
> > > > > > > "response":{"numFound":1,"start":0,"docs":[
> > > > > > > {
> > > > > > > "id":"1",
> > > > > > > "content":"my-content-value",
> > > > > > > "*content-type*":"text/plain"}]
> > > > > > > }}
> > > > > > >
> > > > > > > when i want
> > > > > > >
> > > > > > > my-content-value
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev <mk...@apache.org>
> > > wrote:
> > > > > > >
> > > > > > >> Lee, from this description I don see why it can't be addressed
> > by
> > > > > > fl,rows
> > > > > > >> params. What makes it different form the typical Solr usage?
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll <
> > > > > > >> lee.a.carroll@googlemail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Sure, we want to return a single field's value for the top
> > > > matching
> > > > > > >> > document for a given query. Bare content rather than a full
> > > search
> > > > > > >> result
> > > > > > >> > listing.
> > > > > > >> >
> > > > > > >> > To be concrete:
> > > > > > >> >
> > > > > > >> > For a schema of fields id [unique key],
> > > > > content[stored],content-type[
> > > > > > >> > stored]
> > > > > > >> > For a request:
> > > > > > >> >
> > > > > > >> > 1. Request URL:
> > > > > > >> > https://localhost/solr/content?q=id:1
> > > > > > >> > 2. Request Method:
> > > > > > >> > GET
> > > > > > >> >
> > > > > > >> > We get a response
> > > > > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type:
> > > [content-type
> > > > > > value]
> > > > > > >> >
> > > > > > >> > and the body to be the raw value of content
> > > > > > >> >
> > > > > > >> > In short clients consume directly the most relevant
> "content"
> > > > > returned
> > > > > > >> from
> > > > > > >> > solr queries they construct.
> > > > > > >> >
> > > > > > >> > Naively I've implemented a subclass of RawResponseWriter
> which
> > > > takes
> > > > > > the
> > > > > > >> > first docs values and adds them to the appended "content"
> > > stream.
> > > > > > >> Should I
> > > > > > >> > selectively add the content stream depending on if this is
> the
> > > > final
> > > > > > >> > aggregation of cloud results (and provide a base class
> writer
> > to
> > > > act
> > > > > > if
> > > > > > >> > not), if so how do I know its the final aggregation. Or is
> > > adding
> > > > > the
> > > > > > >> > content stream within the response writer a bad idea. Should
> > > that
> > > > be
> > > > > > >> being
> > > > > > >> > added to the response somewhere else?
> > > > > > >> >
> > > > > > >> > Failing all of the above is asking about response writer an
> X
> > /
> > > Y
> > > > > > >> problem.
> > > > > > >> > Is their a better way to achieve the above. I'd looked at
> > > > > transforming
> > > > > > >> > response xml but that seemed not to offer a complete bare
> > slate.
> > > > > > >> >
> > > > > > >> > Cheers Lee C
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On 17 April 2018 at 21:36, Mikhail Khludnev <
> mkhl@apache.org>
> > > > > wrote:
> > > > > > >> >
> > > > > > >> > > In distributed search response writer is used twice
> > > > > > >> > > https://lucene.apache.org/solr/guide/7_1/distributed-
> > > > > requests.html
> > > > > > >> > > once slave node that's where response writer yields "json"
> > > > content
> > > > > > >> and it
> > > > > > >> > > upset aggregator node which is expect only javabin.
> > > > > > >> > > I hardly can comment on rrw, it's probably used for
> > responding
> > > > > > >> separate
> > > > > > >> > > files in distrib=false mode.
> > > > > > >> > > You can start from describing why you need to create own
> > > > response
> > > > > > >> writer.
> > > > > > >> > >
> > > > > > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <
> > > > > > >> > lee.a.carroll@googlemail.com
> > > > > > >> > > >
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Ok. My expectation was the response writer would not be
> > used
> > > > > until
> > > > > > >> the
> > > > > > >> > > > final serialization of the result. If my response writer
> > > > breaks
> > > > > > the
> > > > > > >> > > > response writer contract, exactly the way
> > rawResponseWriter
> > > > does
> > > > > > and
> > > > > > >> > just
> > > > > > >> > > > out puts a filed value how does that work? Does
> > > > > rawResponseWriter
> > > > > > >> > support
> > > > > > >> > > > cloud mode?
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev <
> > > mkhl@apache.org>
> > > > > > >> wrote:
> > > > > > >> > > >
> > > > > > >> > > > > That's what should happen.
> > > > > > >> > > > >
> > > > > > >> > > > > Expected mime type application/octet-stream but got
> > > > > > >> application/json.
> > > > > > >> > > > >
> > > > > > >> > > > > Distributed search coordinator expect to merge slave
> > > > responses
> > > > > > in
> > > > > > >> > > javabin
> > > > > > >> > > > > format. But slave's wt indicated json.
> > > > > > >> > > > > As far as I know only javabin might be used to
> > distributed
> > > > > > search
> > > > > > >> > > > > underneath. Coordinator itself might yield json.
> > > > > > >> > > > >
> > > > > > >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
> > > > > > >> > > > lee.a.carroll@googlemail.com
> > > > > > >> > > > > >
> > > > > > >> > > > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Sure
> > > > > > >> > > > > >
> > > > > > >> > > > > > with 1 shard 1 replica this request works fine
> > > > > > >> > > > > >
> > > > > > >> > > > > > 1. Request URL:
> > > > > > >> > > > > > http://localhost:8983/solr/images/image?q=id:1
> > > > > > >> > > > > > 2. Request Method:
> > > > > > >> > > > > > GET
> > > > > > >> > > > > > 3. Status Code:
> > > > > > >> > > > > > 200 OK
> > > > > > >> > > > > >
> > > > > > >> > > > > > logs are clean
> > > > > > >> > > > > >
> > > > > > >> > > > > > with 2 shards 2 replicas the same request fails and
> in
> > > the
> > > > > > logs
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2
> > > > > > r:core_node7
> > > > > > >> > > > > > x:images_shard2_replica_n4]
> > > org.apache.solr.core.SolrCore;
> > > > > > >> > > > > > [images_shard2_replica_n4] webapp=/solr path=/image
> > > > > > >> > > > > > params={df=text&distrib=false&
> > qt=/image&fl=id&fl=score&
> > > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > > > > >> > > > > > http://10.224.30.207:8983/
> > > solr/images_shard2_replica_n4/
> > > > > > >> > > > > > |http://10.224.30.207:7574/
> > > solr/images_shard2_replica_n6/
> > > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
> > > > > 1523971232039&isShard=true&wt=
> > > > > > >> > javabin}
> > > > > > >> > > > > > hits=0 status=0 QTime=0
> > > > > > >> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1
> > > > > > r:core_node3
> > > > > > >> > > > > > x:images_shard1_replica_n1]
> > > org.apache.solr.common.SolrExc
> > > > > > >> eption;
> > > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
> > > > > > >> > > RemoteSolrException:
> > > > > > >> > > > > > Error
> > > > > > >> > > > > > from server at http://10.224.30.207:8983/
> > > > > > >> > > solr/images_shard2_replica_n4
> > > > > > >> > > > :
> > > > > > >> > > > > > Expected mime type application/octet-stream but got
> > > > > > >> > application/json.
> > > > > > >> > > > > > at
> > > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
> > > > > > >> > > > > > executeMethod(HttpSolrClient.java:607)
> > > > > > >> > > > > > at
> > > > > > >> > > > > > org.apache.solr.client.solrj.
> > > impl.HttpSolrClient.request(
> > > > > > >> > > > > > HttpSolrClient.java:255)
> > > > > > >> > > > > > at
> > > > > > >> > > > > > org.apache.solr.client.solrj.
> > > impl.HttpSolrClient.request(
> > > > > > >> > > > > > HttpSolrClient.java:244)
> > > > > > >> > > > > > at
> > > > > > >> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > > > > > >> > > > > > doRequest(LBHttpSolrClient.java:483)
> > > > > > >> > > > > > at
> > > > > > >> > > > > > org.apache.solr.client.solrj.
> > > > impl.LBHttpSolrClient.request(
> > > > > > >> > > > > > LBHttpSolrClient.java:413)
> > > > > > >> > > > > > at
> > > > > > >> > > > > > org.apache.solr.handler.component.
> > > > HttpShardHandlerFactory.
> > > > > > >> > > > > > makeLoadBalancedRequest(
> HttpShardHandlerFactory.java:
> > > 273)
> > > > > > >> > > > > > at
> > > > > > >> > > > > > org.apache.solr.handler.component.HttpShardHandler.
> > > > > > >> > lambda$submit$0(
> > > > > > >> > > > > > HttpShardHandler.java:175)
> > > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
> > > > java:266)
> > > > > > >> > > > > > at java.util.concurrent.Executors$RunnableAdapter.
> > > > > > >> > > > > call(Executors.java:511)
> > > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
> > > > java:266)
> > > > > > >> > > > > > at
> > > > > > >> > > > > > com.codahale.metrics.InstrumentedExecutorService$
> > > > > > >> > > > > InstrumentedRunnable.run(
> > > > > > >> > > > > > InstrumentedExecutorService.java:176)
> > > > > > >> > > > > > at
> > > > > > >> > > > > > org.apache.solr.common.util.ExecutorUtil$
> > > > > > >> > MDCAwareThreadPoolExecutor.
> > > > > > >> > > > > > lambda$execute$0(ExecutorUtil.java:188)
> > > > > > >> > > > > > at
> > > > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > >> > > > > > ThreadPoolExecutor.java:1142)
> > > > > > >> > > > > > at
> > > > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > >> > > > > > ThreadPoolExecutor.java:617)
> > > > > > >> > > > > > at java.lang.Thread.run(Thread.java:745)
> > > > > > >> > > > > >
> > > > > > >> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1
> > > > > > r:core_node3
> > > > > > >> > > > > > x:images_shard1_replica_n1]
> > > org.apache.solr.core.SolrCore;
> > > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> > > > > > >> > params={q=id:1}
> > > > > > >> > > > > > status=200 QTime=17
> > > > > > >> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1
> > > > > > r:core_node3
> > > > > > >> > > > > > x:images_shard1_replica_n1]
> > > org.apache.solr.core.SolrCore;
> > > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> > > > > > >> > > > > > params={df=text&distrib=false&
> > qt=/image&fl=id&fl=score&
> > > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > > > > >> > > > > > http://10.224.30.207:8983/
> > > solr/images_shard1_replica_n1/
> > > > > > >> > > > > > |http://10.224.30.207:7574/
> > > solr/images_shard1_replica_n2/
> > > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
> > > > > 1523971232039&isShard=true&wt=
> > > > > > >> > javabin}
> > > > > > >> > > > > > hits=1 status=0 QTime=2
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > I've implemented getcontenttype simply as
> > > > > > >> > > > > >
> > > > > > >> > > > > > @Override
> > > > > > >> > > > > > public String getContentType(SolrQueryRequest
> > request,
> > > > > > >> > > > > > SolrQueryResponse response) {
> > > > > > >> > > > > >
> > > > > > >> > > > > > return "application/json;charset=utf-8";
> > > > > > >> > > > > > }
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev <
> > > > > mkhl@apache.org>
> > > > > > >> > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Lee,
> > > > > > >> > > > > > > It's worth to send a stacktrace for such kind of
> > > > > inquiries.
> > > > > > >> > > > > > > I guess it goes from QueryComponent.mergeIds() or
> > so.
> > > > > Shard
> > > > > > >> > > response
> > > > > > >> > > > > > should
> > > > > > >> > > > > > > contains <uniqueKey> from schema.xml field.
> > > > > > >> > > > > > > I encounter something like this while
> > troubleshooting
> > > > > > >> > > > > > > https://lucene.apache.org/
> > > solr/guide/6_6/transforming-
> > > > > > >> > > > > > > result-documents.html#
> TransformingResultDocuments-
> > > > > > >> > > > > > > CoresandCollectionsinSolrCloud
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
> > > > > > >> > > > > > lee.a.carroll@googlemail.com
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > I've created a custom response writer which
> > extends
> > > > > > >> > > > > RawResponseWriter.
> > > > > > >> > > > > > > The
> > > > > > >> > > > > > > > basic operation is to output a single field
> value
> > > from
> > > > > the
> > > > > > >> top
> > > > > > >> > > > > matching
> > > > > > >> > > > > > > doc
> > > > > > >> > > > > > > > as the entire response. This works when shards
> = 1
> > > but
> > > > > > fails
> > > > > > >> > when
> > > > > > >> > > > > > shards
> > > > > > >> > > > > > > > are greater than 1.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > I throw an error if the field in question is
> > missing
> > > > > from
> > > > > > >> the
> > > > > > >> > top
> > > > > > >> > > > > doc.
> > > > > > >> > > > > > > This
> > > > > > >> > > > > > > > happens when individual shards are being
> searched
> > > and
> > > > > only
> > > > > > >> id
> > > > > > >> > and
> > > > > > >> > > > > score
> > > > > > >> > > > > > > are
> > > > > > >> > > > > > > > returned. I'm sure I've committed a basic error.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Lee C
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > --
> > > > > > >> > > > > > > Sincerely yours
> > > > > > >> > > > > > > Mikhail Khludnev
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > --
> > > > > > >> > > > > Sincerely yours
> > > > > > >> > > > > Mikhail Khludnev
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > --
> > > > > > >> > > Sincerely yours
> > > > > > >> > > Mikhail Khludnev
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> Sincerely yours
> > > > > > >> Mikhail Khludnev
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sincerely yours
> > > > > Mikhail Khludnev
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > >
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Mikhail Khludnev <mk...@apache.org>.
what if you put it into "defaults"?
On Thu, Apr 19, 2018 at 8:42 PM, Lee Carroll <le...@googlemail.com>
wrote:
> Hi,
>
> I rewrote all of my tests to use SolrCloudTestCase rather than
> SolrTestCaseJ4
> and was able to replicate the responsewriter issue and debug with a sharded
> collection. It turned out the issue was not with my response writer really
> but rather my config.
>
> <requestHandler name="/content" class="solr.SearchHandler">
> ....
>
> <lst name="invariants">
> <str name="wt">content</str>
> </lst>
>
> </requestHandler>
>
> In cloud mode having wt as an invariant breaks the collation of results
> from shards. Now I'm sure this is a common mistake which I've repeated
> (blush) but I do sort of want to actually implement my request handler in
> this way. Is their a way to have a request handler support a single
> response writer but still work in cloud mode ?
>
> Could this be considered a bug ?
>
> Lee C
>
> On 18 April 2018 at 13:13, Mikhail Khludnev <mk...@apache.org> wrote:
>
> > Injecting headers might require deeper customisation up to establishing
> own
> > filter or so.
> > Speaking regarding your own WT, there might be some issues because
> usually
> > it's not a big deal to use one wt for responding user query like (wt=csv)
> > and wt=javabin in internal communication between aggregator and slaves
> like
> > it happens in wt=csv query.
> >
> > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll <
> lee.a.carroll@googlemail.com
> > >
> > wrote:
> >
> > > Inventive. I need to control content-type of the response from the
> > document
> > > field value. I have the actual content field and the content-type field
> > to
> > > use configured in the response writer. I've just noticed that the xslt
> > > transformer allows you to do this but not controlled by document
> values.
> > I
> > > may also need to set some headers based on content-type and perhaps
> > content
> > > size, accept ranges comes to mind. Although I might be getting ahead of
> > > myself.
> > >
> > >
> > >
> > > On 18 April 2018 at 12:05, Mikhail Khludnev <mk...@apache.org> wrote:
> > >
> > > > well ..
> > > > what if
> > > > http://localhost:8983/solr/images/select?fl=content&q=id:
> > > 1&start=1&wt=csv&
> > > > csv.separator=&csv.encapsulator&csv.null=null
> > > > ?
> > > >
> > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll <
> > > lee.a.carroll@googlemail.com
> > > > >
> > > > wrote:
> > > >
> > > > > sorry cut n paste error i'd get
> > > > >
> > > > > {
> > > > > "responseHeader":{
> > > > > "zkConnected":true,
> > > > > "status":0,
> > > > > "QTime":0,
> > > > > "params":{
> > > > > "q":"*:*",
> > > > > "fl":"content",
> > > > > "rows":"1"}},
> > > > > "response":{"numFound":1,"start":0,"docs":[
> > > > > {
> > > > > "content":"my-content-value"}]
> > > > > }}
> > > > >
> > > > >
> > > > > but you get my point
> > > > >
> > > > >
> > > > >
> > > > > On 18 April 2018 at 11:13, Lee Carroll <
> lee.a.carroll@googlemail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > for http://localhost:8983/solr/images/select?fl=content&q=id:
> > > 1&start=1
> > > > > >
> > > > > > I'd get
> > > > > >
> > > > > > {
> > > > > > "responseHeader":{
> > > > > > "zkConnected":true,
> > > > > > "status":0,
> > > > > > "QTime":1,
> > > > > > "params":{
> > > > > > "q":"*:*",
> > > > > > "_":"1524046333220"}},
> > > > > > "response":{"numFound":1,"start":0,"docs":[
> > > > > > {
> > > > > > "id":"1",
> > > > > > "content":"my-content-value",
> > > > > > "*content-type*":"text/plain"}]
> > > > > > }}
> > > > > >
> > > > > > when i want
> > > > > >
> > > > > > my-content-value
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev <mk...@apache.org>
> > wrote:
> > > > > >
> > > > > >> Lee, from this description I don see why it can't be addressed
> by
> > > > > fl,rows
> > > > > >> params. What makes it different form the typical Solr usage?
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll <
> > > > > >> lee.a.carroll@googlemail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Sure, we want to return a single field's value for the top
> > > matching
> > > > > >> > document for a given query. Bare content rather than a full
> > search
> > > > > >> result
> > > > > >> > listing.
> > > > > >> >
> > > > > >> > To be concrete:
> > > > > >> >
> > > > > >> > For a schema of fields id [unique key],
> > > > content[stored],content-type[
> > > > > >> > stored]
> > > > > >> > For a request:
> > > > > >> >
> > > > > >> > 1. Request URL:
> > > > > >> > https://localhost/solr/content?q=id:1
> > > > > >> > 2. Request Method:
> > > > > >> > GET
> > > > > >> >
> > > > > >> > We get a response
> > > > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type:
> > [content-type
> > > > > value]
> > > > > >> >
> > > > > >> > and the body to be the raw value of content
> > > > > >> >
> > > > > >> > In short clients consume directly the most relevant "content"
> > > > returned
> > > > > >> from
> > > > > >> > solr queries they construct.
> > > > > >> >
> > > > > >> > Naively I've implemented a subclass of RawResponseWriter which
> > > takes
> > > > > the
> > > > > >> > first docs values and adds them to the appended "content"
> > stream.
> > > > > >> Should I
> > > > > >> > selectively add the content stream depending on if this is the
> > > final
> > > > > >> > aggregation of cloud results (and provide a base class writer
> to
> > > act
> > > > > if
> > > > > >> > not), if so how do I know its the final aggregation. Or is
> > adding
> > > > the
> > > > > >> > content stream within the response writer a bad idea. Should
> > that
> > > be
> > > > > >> being
> > > > > >> > added to the response somewhere else?
> > > > > >> >
> > > > > >> > Failing all of the above is asking about response writer an X
> /
> > Y
> > > > > >> problem.
> > > > > >> > Is their a better way to achieve the above. I'd looked at
> > > > transforming
> > > > > >> > response xml but that seemed not to offer a complete bare
> slate.
> > > > > >> >
> > > > > >> > Cheers Lee C
> > > > > >> >
> > > > > >> >
> > > > > >> > On 17 April 2018 at 21:36, Mikhail Khludnev <mk...@apache.org>
> > > > wrote:
> > > > > >> >
> > > > > >> > > In distributed search response writer is used twice
> > > > > >> > > https://lucene.apache.org/solr/guide/7_1/distributed-
> > > > requests.html
> > > > > >> > > once slave node that's where response writer yields "json"
> > > content
> > > > > >> and it
> > > > > >> > > upset aggregator node which is expect only javabin.
> > > > > >> > > I hardly can comment on rrw, it's probably used for
> responding
> > > > > >> separate
> > > > > >> > > files in distrib=false mode.
> > > > > >> > > You can start from describing why you need to create own
> > > response
> > > > > >> writer.
> > > > > >> > >
> > > > > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <
> > > > > >> > lee.a.carroll@googlemail.com
> > > > > >> > > >
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > Ok. My expectation was the response writer would not be
> used
> > > > until
> > > > > >> the
> > > > > >> > > > final serialization of the result. If my response writer
> > > breaks
> > > > > the
> > > > > >> > > > response writer contract, exactly the way
> rawResponseWriter
> > > does
> > > > > and
> > > > > >> > just
> > > > > >> > > > out puts a filed value how does that work? Does
> > > > rawResponseWriter
> > > > > >> > support
> > > > > >> > > > cloud mode?
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev <
> > mkhl@apache.org>
> > > > > >> wrote:
> > > > > >> > > >
> > > > > >> > > > > That's what should happen.
> > > > > >> > > > >
> > > > > >> > > > > Expected mime type application/octet-stream but got
> > > > > >> application/json.
> > > > > >> > > > >
> > > > > >> > > > > Distributed search coordinator expect to merge slave
> > > responses
> > > > > in
> > > > > >> > > javabin
> > > > > >> > > > > format. But slave's wt indicated json.
> > > > > >> > > > > As far as I know only javabin might be used to
> distributed
> > > > > search
> > > > > >> > > > > underneath. Coordinator itself might yield json.
> > > > > >> > > > >
> > > > > >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
> > > > > >> > > > lee.a.carroll@googlemail.com
> > > > > >> > > > > >
> > > > > >> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Sure
> > > > > >> > > > > >
> > > > > >> > > > > > with 1 shard 1 replica this request works fine
> > > > > >> > > > > >
> > > > > >> > > > > > 1. Request URL:
> > > > > >> > > > > > http://localhost:8983/solr/images/image?q=id:1
> > > > > >> > > > > > 2. Request Method:
> > > > > >> > > > > > GET
> > > > > >> > > > > > 3. Status Code:
> > > > > >> > > > > > 200 OK
> > > > > >> > > > > >
> > > > > >> > > > > > logs are clean
> > > > > >> > > > > >
> > > > > >> > > > > > with 2 shards 2 replicas the same request fails and in
> > the
> > > > > logs
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2
> > > > > r:core_node7
> > > > > >> > > > > > x:images_shard2_replica_n4]
> > org.apache.solr.core.SolrCore;
> > > > > >> > > > > > [images_shard2_replica_n4] webapp=/solr path=/image
> > > > > >> > > > > > params={df=text&distrib=false&
> qt=/image&fl=id&fl=score&
> > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > > > >> > > > > > http://10.224.30.207:8983/
> > solr/images_shard2_replica_n4/
> > > > > >> > > > > > |http://10.224.30.207:7574/
> > solr/images_shard2_replica_n6/
> > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
> > > > 1523971232039&isShard=true&wt=
> > > > > >> > javabin}
> > > > > >> > > > > > hits=0 status=0 QTime=0
> > > > > >> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1
> > > > > r:core_node3
> > > > > >> > > > > > x:images_shard1_replica_n1]
> > org.apache.solr.common.SolrExc
> > > > > >> eption;
> > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
> > > > > >> > > RemoteSolrException:
> > > > > >> > > > > > Error
> > > > > >> > > > > > from server at http://10.224.30.207:8983/
> > > > > >> > > solr/images_shard2_replica_n4
> > > > > >> > > > :
> > > > > >> > > > > > Expected mime type application/octet-stream but got
> > > > > >> > application/json.
> > > > > >> > > > > > at
> > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
> > > > > >> > > > > > executeMethod(HttpSolrClient.java:607)
> > > > > >> > > > > > at
> > > > > >> > > > > > org.apache.solr.client.solrj.
> > impl.HttpSolrClient.request(
> > > > > >> > > > > > HttpSolrClient.java:255)
> > > > > >> > > > > > at
> > > > > >> > > > > > org.apache.solr.client.solrj.
> > impl.HttpSolrClient.request(
> > > > > >> > > > > > HttpSolrClient.java:244)
> > > > > >> > > > > > at
> > > > > >> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > > > > >> > > > > > doRequest(LBHttpSolrClient.java:483)
> > > > > >> > > > > > at
> > > > > >> > > > > > org.apache.solr.client.solrj.
> > > impl.LBHttpSolrClient.request(
> > > > > >> > > > > > LBHttpSolrClient.java:413)
> > > > > >> > > > > > at
> > > > > >> > > > > > org.apache.solr.handler.component.
> > > HttpShardHandlerFactory.
> > > > > >> > > > > > makeLoadBalancedRequest(HttpShardHandlerFactory.java:
> > 273)
> > > > > >> > > > > > at
> > > > > >> > > > > > org.apache.solr.handler.component.HttpShardHandler.
> > > > > >> > lambda$submit$0(
> > > > > >> > > > > > HttpShardHandler.java:175)
> > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
> > > java:266)
> > > > > >> > > > > > at java.util.concurrent.Executors$RunnableAdapter.
> > > > > >> > > > > call(Executors.java:511)
> > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
> > > java:266)
> > > > > >> > > > > > at
> > > > > >> > > > > > com.codahale.metrics.InstrumentedExecutorService$
> > > > > >> > > > > InstrumentedRunnable.run(
> > > > > >> > > > > > InstrumentedExecutorService.java:176)
> > > > > >> > > > > > at
> > > > > >> > > > > > org.apache.solr.common.util.ExecutorUtil$
> > > > > >> > MDCAwareThreadPoolExecutor.
> > > > > >> > > > > > lambda$execute$0(ExecutorUtil.java:188)
> > > > > >> > > > > > at
> > > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > >> > > > > > ThreadPoolExecutor.java:1142)
> > > > > >> > > > > > at
> > > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > >> > > > > > ThreadPoolExecutor.java:617)
> > > > > >> > > > > > at java.lang.Thread.run(Thread.java:745)
> > > > > >> > > > > >
> > > > > >> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1
> > > > > r:core_node3
> > > > > >> > > > > > x:images_shard1_replica_n1]
> > org.apache.solr.core.SolrCore;
> > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> > > > > >> > params={q=id:1}
> > > > > >> > > > > > status=200 QTime=17
> > > > > >> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1
> > > > > r:core_node3
> > > > > >> > > > > > x:images_shard1_replica_n1]
> > org.apache.solr.core.SolrCore;
> > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> > > > > >> > > > > > params={df=text&distrib=false&
> qt=/image&fl=id&fl=score&
> > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > > > >> > > > > > http://10.224.30.207:8983/
> > solr/images_shard1_replica_n1/
> > > > > >> > > > > > |http://10.224.30.207:7574/
> > solr/images_shard1_replica_n2/
> > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
> > > > 1523971232039&isShard=true&wt=
> > > > > >> > javabin}
> > > > > >> > > > > > hits=1 status=0 QTime=2
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > I've implemented getcontenttype simply as
> > > > > >> > > > > >
> > > > > >> > > > > > @Override
> > > > > >> > > > > > public String getContentType(SolrQueryRequest
> request,
> > > > > >> > > > > > SolrQueryResponse response) {
> > > > > >> > > > > >
> > > > > >> > > > > > return "application/json;charset=utf-8";
> > > > > >> > > > > > }
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev <
> > > > mkhl@apache.org>
> > > > > >> > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Lee,
> > > > > >> > > > > > > It's worth to send a stacktrace for such kind of
> > > > inquiries.
> > > > > >> > > > > > > I guess it goes from QueryComponent.mergeIds() or
> so.
> > > > Shard
> > > > > >> > > response
> > > > > >> > > > > > should
> > > > > >> > > > > > > contains <uniqueKey> from schema.xml field.
> > > > > >> > > > > > > I encounter something like this while
> troubleshooting
> > > > > >> > > > > > > https://lucene.apache.org/
> > solr/guide/6_6/transforming-
> > > > > >> > > > > > > result-documents.html#TransformingResultDocuments-
> > > > > >> > > > > > > CoresandCollectionsinSolrCloud
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
> > > > > >> > > > > > lee.a.carroll@googlemail.com
> > > > > >> > > > > > > >
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > I've created a custom response writer which
> extends
> > > > > >> > > > > RawResponseWriter.
> > > > > >> > > > > > > The
> > > > > >> > > > > > > > basic operation is to output a single field value
> > from
> > > > the
> > > > > >> top
> > > > > >> > > > > matching
> > > > > >> > > > > > > doc
> > > > > >> > > > > > > > as the entire response. This works when shards = 1
> > but
> > > > > fails
> > > > > >> > when
> > > > > >> > > > > > shards
> > > > > >> > > > > > > > are greater than 1.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > I throw an error if the field in question is
> missing
> > > > from
> > > > > >> the
> > > > > >> > top
> > > > > >> > > > > doc.
> > > > > >> > > > > > > This
> > > > > >> > > > > > > > happens when individual shards are being searched
> > and
> > > > only
> > > > > >> id
> > > > > >> > and
> > > > > >> > > > > score
> > > > > >> > > > > > > are
> > > > > >> > > > > > > > returned. I'm sure I've committed a basic error.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Lee C
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > --
> > > > > >> > > > > > > Sincerely yours
> > > > > >> > > > > > > Mikhail Khludnev
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > --
> > > > > >> > > > > Sincerely yours
> > > > > >> > > > > Mikhail Khludnev
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > --
> > > > > >> > > Sincerely yours
> > > > > >> > > Mikhail Khludnev
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Sincerely yours
> > > > > >> Mikhail Khludnev
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sincerely yours
> > > > Mikhail Khludnev
> > > >
> > >
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> >
>
--
Sincerely yours
Mikhail Khludnev
Re: custom response writer which extends RawResponseWriter fails
when shards > 1
Posted by Chris Hostetter <ho...@fucit.org>.
Not that i can think of -- the existing API is really designed with the
focus that the the ResponseWriter needs to specify the Content-Type prior
to "writting" any bytes so Solr can do the best job possible "streaming"
the data over the wire ... if you need to "pre-process" a lot of the
response data before you know what you want to stream back, you have to do
that in the getContentType() method .. but you can always store that
pre-processed result in the SolrQueryRequest.getContext() and then write
it out later.
(you could in theory do a ton of processing to prepare a big byte[] of all
the data you want to send out as part of getContentType(), and then your
write() method could be a single line of code)
: Date: Sun, 22 Apr 2018 15:49:30 +0100
: From: Lee Carroll <le...@googlemail.com>
: Reply-To: solr-user@lucene.apache.org
: To: solr-user@lucene.apache.org
: Subject: Re: custom response writer which extends RawResponseWriter fails when
: shards > 1
:
: Hi,
: I've ended up processing the doclist in the response twice. Once in the
: write method and once in getContent. Its a bit inefficient but i'm only
: looking at top doc each time so probably ok.
: Is their a better way to do this ?
:
: Cheers lee C
:
: On 22 April 2018 at 13:26, Lee Carroll <le...@googlemail.com> wrote:
:
: > Hi,
: > That works a treat. The raw response writer has a configurable base class
: > which executes when no content stream is present in the response so I just
: > delegate to that. I do still have an issue with writing content type on the
: > http response from a value in the top document however. Although I do have
: > a getContentType method which returns a value from a field in the top
: > document this is called before the response writer has performed its write
: > method.
: >
: > Whats the best way to set response headers using values from documents in
: > the search result? In particular Content-Type.
: >
: > Cheers Lee C
: >
: > On 20 April 2018 at 20:03, Chris Hostetter <ho...@fucit.org>
: > wrote:
: >
: >>
: >> Invariant really means "invariant" ... nothing can change them
: >>
: >> In the case of "wt" this may seem weird and unhelpful, but the code that
: >> handles defaults/appends/invariants is ignorant of what the params are.
: >>
: >> Since your writting custom code anyway, my suggestion would be that
: >> perhaps you could make your custom ResponseWriter delegate to the javabin
: >> responsewriter if/when you see that this is an "isShard=true" request?
: >>
: >>
: >>
: >> : Date: Thu, 19 Apr 2018 18:42:58 +0100
: >> : From: Lee Carroll <le...@googlemail.com>
: >> : Reply-To: solr-user@lucene.apache.org
: >> : To: solr-user@lucene.apache.org
: >> : Subject: Re: custom response writer which extends RawResponseWriter
: >> fails when
: >> : shards > 1
: >> :
: >> : Hi,
: >> :
: >> : I rewrote all of my tests to use SolrCloudTestCase rather than
: >> SolrTestCaseJ4
: >> : and was able to replicate the responsewriter issue and debug with a
: >> sharded
: >> : collection. It turned out the issue was not with my response writer
: >> really
: >> : but rather my config.
: >> :
: >> : <requestHandler name="/content" class="solr.SearchHandler">
: >> : ....
: >> :
: >> : <lst name="invariants">
: >> : <str name="wt">content</str>
: >> : </lst>
: >> :
: >> : </requestHandler>
: >> :
: >> : In cloud mode having wt as an invariant breaks the collation of results
: >> : from shards. Now I'm sure this is a common mistake which I've repeated
: >> : (blush) but I do sort of want to actually implement my request handler
: >> in
: >> : this way. Is their a way to have a request handler support a single
: >> : response writer but still work in cloud mode ?
: >> :
: >> : Could this be considered a bug ?
: >> :
: >> : Lee C
: >> :
: >> : On 18 April 2018 at 13:13, Mikhail Khludnev <mk...@apache.org> wrote:
: >> :
: >> : > Injecting headers might require deeper customisation up to
: >> establishing own
: >> : > filter or so.
: >> : > Speaking regarding your own WT, there might be some issues because
: >> usually
: >> : > it's not a big deal to use one wt for responding user query like
: >> (wt=csv)
: >> : > and wt=javabin in internal communication between aggregator and
: >> slaves like
: >> : > it happens in wt=csv query.
: >> : >
: >> : > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll <
: >> lee.a.carroll@googlemail.com
: >> : > >
: >> : > wrote:
: >> : >
: >> : > > Inventive. I need to control content-type of the response from the
: >> : > document
: >> : > > field value. I have the actual content field and the content-type
: >> field
: >> : > to
: >> : > > use configured in the response writer. I've just noticed that the
: >> xslt
: >> : > > transformer allows you to do this but not controlled by document
: >> values.
: >> : > I
: >> : > > may also need to set some headers based on content-type and perhaps
: >> : > content
: >> : > > size, accept ranges comes to mind. Although I might be getting
: >> ahead of
: >> : > > myself.
: >> : > >
: >> : > >
: >> : > >
: >> : > > On 18 April 2018 at 12:05, Mikhail Khludnev <mk...@apache.org>
: >> wrote:
: >> : > >
: >> : > > > well ..
: >> : > > > what if
: >> : > > > http://localhost:8983/solr/images/select?fl=content&q=id:
: >> : > > 1&start=1&wt=csv&
: >> : > > > csv.separator=&csv.encapsulator&csv.null=null
: >> : > > > ?
: >> : > > >
: >> : > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll <
: >> : > > lee.a.carroll@googlemail.com
: >> : > > > >
: >> : > > > wrote:
: >> : > > >
: >> : > > > > sorry cut n paste error i'd get
: >> : > > > >
: >> : > > > > {
: >> : > > > > "responseHeader":{
: >> : > > > > "zkConnected":true,
: >> : > > > > "status":0,
: >> : > > > > "QTime":0,
: >> : > > > > "params":{
: >> : > > > > "q":"*:*",
: >> : > > > > "fl":"content",
: >> : > > > > "rows":"1"}},
: >> : > > > > "response":{"numFound":1,"start":0,"docs":[
: >> : > > > > {
: >> : > > > > "content":"my-content-value"}]
: >> : > > > > }}
: >> : > > > >
: >> : > > > >
: >> : > > > > but you get my point
: >> : > > > >
: >> : > > > >
: >> : > > > >
: >> : > > > > On 18 April 2018 at 11:13, Lee Carroll <
: >> lee.a.carroll@googlemail.com
: >> : > >
: >> : > > > > wrote:
: >> : > > > >
: >> : > > > > > for http://localhost:8983/solr/images/select?fl=content&q=id:
: >> : > > 1&start=1
: >> : > > > > >
: >> : > > > > > I'd get
: >> : > > > > >
: >> : > > > > > {
: >> : > > > > > "responseHeader":{
: >> : > > > > > "zkConnected":true,
: >> : > > > > > "status":0,
: >> : > > > > > "QTime":1,
: >> : > > > > > "params":{
: >> : > > > > > "q":"*:*",
: >> : > > > > > "_":"1524046333220"}},
: >> : > > > > > "response":{"numFound":1,"start":0,"docs":[
: >> : > > > > > {
: >> : > > > > > "id":"1",
: >> : > > > > > "content":"my-content-value",
: >> : > > > > > "*content-type*":"text/plain"}]
: >> : > > > > > }}
: >> : > > > > >
: >> : > > > > > when i want
: >> : > > > > >
: >> : > > > > > my-content-value
: >> : > > > > >
: >> : > > > > >
: >> : > > > > >
: >> : > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev <mk...@apache.org>
: >> : > wrote:
: >> : > > > > >
: >> : > > > > >> Lee, from this description I don see why it can't be
: >> addressed by
: >> : > > > > fl,rows
: >> : > > > > >> params. What makes it different form the typical Solr usage?
: >> : > > > > >>
: >> : > > > > >>
: >> : > > > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll <
: >> : > > > > >> lee.a.carroll@googlemail.com>
: >> : > > > > >> wrote:
: >> : > > > > >>
: >> : > > > > >> > Sure, we want to return a single field's value for the top
: >> : > > matching
: >> : > > > > >> > document for a given query. Bare content rather than a full
: >> : > search
: >> : > > > > >> result
: >> : > > > > >> > listing.
: >> : > > > > >> >
: >> : > > > > >> > To be concrete:
: >> : > > > > >> >
: >> : > > > > >> > For a schema of fields id [unique key],
: >> : > > > content[stored],content-type[
: >> : > > > > >> > stored]
: >> : > > > > >> > For a request:
: >> : > > > > >> >
: >> : > > > > >> > 1. Request URL:
: >> : > > > > >> > https://localhost/solr/content?q=id:1
: >> : > > > > >> > 2. Request Method:
: >> : > > > > >> > GET
: >> : > > > > >> >
: >> : > > > > >> > We get a response
: >> : > > > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type:
: >> : > [content-type
: >> : > > > > value]
: >> : > > > > >> >
: >> : > > > > >> > and the body to be the raw value of content
: >> : > > > > >> >
: >> : > > > > >> > In short clients consume directly the most relevant
: >> "content"
: >> : > > > returned
: >> : > > > > >> from
: >> : > > > > >> > solr queries they construct.
: >> : > > > > >> >
: >> : > > > > >> > Naively I've implemented a subclass of RawResponseWriter
: >> which
: >> : > > takes
: >> : > > > > the
: >> : > > > > >> > first docs values and adds them to the appended "content"
: >> : > stream.
: >> : > > > > >> Should I
: >> : > > > > >> > selectively add the content stream depending on if this is
: >> the
: >> : > > final
: >> : > > > > >> > aggregation of cloud results (and provide a base class
: >> writer to
: >> : > > act
: >> : > > > > if
: >> : > > > > >> > not), if so how do I know its the final aggregation. Or is
: >> : > adding
: >> : > > > the
: >> : > > > > >> > content stream within the response writer a bad idea.
: >> Should
: >> : > that
: >> : > > be
: >> : > > > > >> being
: >> : > > > > >> > added to the response somewhere else?
: >> : > > > > >> >
: >> : > > > > >> > Failing all of the above is asking about response writer
: >> an X /
: >> : > Y
: >> : > > > > >> problem.
: >> : > > > > >> > Is their a better way to achieve the above. I'd looked at
: >> : > > > transforming
: >> : > > > > >> > response xml but that seemed not to offer a complete bare
: >> slate.
: >> : > > > > >> >
: >> : > > > > >> > Cheers Lee C
: >> : > > > > >> >
: >> : > > > > >> >
: >> : > > > > >> > On 17 April 2018 at 21:36, Mikhail Khludnev <
: >> mkhl@apache.org>
: >> : > > > wrote:
: >> : > > > > >> >
: >> : > > > > >> > > In distributed search response writer is used twice
: >> : > > > > >> > > https://lucene.apache.org/solr/guide/7_1/distributed-
: >> : > > > requests.html
: >> : > > > > >> > > once slave node that's where response writer yields
: >> "json"
: >> : > > content
: >> : > > > > >> and it
: >> : > > > > >> > > upset aggregator node which is expect only javabin.
: >> : > > > > >> > > I hardly can comment on rrw, it's probably used for
: >> responding
: >> : > > > > >> separate
: >> : > > > > >> > > files in distrib=false mode.
: >> : > > > > >> > > You can start from describing why you need to create own
: >> : > > response
: >> : > > > > >> writer.
: >> : > > > > >> > >
: >> : > > > > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <
: >> : > > > > >> > lee.a.carroll@googlemail.com
: >> : > > > > >> > > >
: >> : > > > > >> > > wrote:
: >> : > > > > >> > >
: >> : > > > > >> > > > Ok. My expectation was the response writer would not
: >> be used
: >> : > > > until
: >> : > > > > >> the
: >> : > > > > >> > > > final serialization of the result. If my response
: >> writer
: >> : > > breaks
: >> : > > > > the
: >> : > > > > >> > > > response writer contract, exactly the way
: >> rawResponseWriter
: >> : > > does
: >> : > > > > and
: >> : > > > > >> > just
: >> : > > > > >> > > > out puts a filed value how does that work? Does
: >> : > > > rawResponseWriter
: >> : > > > > >> > support
: >> : > > > > >> > > > cloud mode?
: >> : > > > > >> > > >
: >> : > > > > >> > > >
: >> : > > > > >> > > >
: >> : > > > > >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev <
: >> : > mkhl@apache.org>
: >> : > > > > >> wrote:
: >> : > > > > >> > > >
: >> : > > > > >> > > > > That's what should happen.
: >> : > > > > >> > > > >
: >> : > > > > >> > > > > Expected mime type application/octet-stream but got
: >> : > > > > >> application/json.
: >> : > > > > >> > > > >
: >> : > > > > >> > > > > Distributed search coordinator expect to merge slave
: >> : > > responses
: >> : > > > > in
: >> : > > > > >> > > javabin
: >> : > > > > >> > > > > format. But slave's wt indicated json.
: >> : > > > > >> > > > > As far as I know only javabin might be used to
: >> distributed
: >> : > > > > search
: >> : > > > > >> > > > > underneath. Coordinator itself might yield json.
: >> : > > > > >> > > > >
: >> : > > > > >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
: >> : > > > > >> > > > lee.a.carroll@googlemail.com
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > wrote:
: >> : > > > > >> > > > >
: >> : > > > > >> > > > > > Sure
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > > with 1 shard 1 replica this request works fine
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > > 1. Request URL:
: >> : > > > > >> > > > > > http://localhost:8983/solr/images/image?q=id:1
: >> : > > > > >> > > > > > 2. Request Method:
: >> : > > > > >> > > > > > GET
: >> : > > > > >> > > > > > 3. Status Code:
: >> : > > > > >> > > > > > 200 OK
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > > logs are clean
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > > with 2 shards 2 replicas the same request fails
: >> and in
: >> : > the
: >> : > > > > logs
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2
: >> : > > > > r:core_node7
: >> : > > > > >> > > > > > x:images_shard2_replica_n4]
: >> : > org.apache.solr.core.SolrCore;
: >> : > > > > >> > > > > > [images_shard2_replica_n4] webapp=/solr
: >> path=/image
: >> : > > > > >> > > > > > params={df=text&distrib=false&
: >> qt=/image&fl=id&fl=score&
: >> : > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
: >> : > > > > >> > > > > > http://10.224.30.207:8983/
: >> : > solr/images_shard2_replica_n4/
: >> : > > > > >> > > > > > |http://10.224.30.207:7574/
: >> : > solr/images_shard2_replica_n6/
: >> : > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
: >> : > > > 1523971232039&isShard=true&wt=
: >> : > > > > >> > javabin}
: >> : > > > > >> > > > > > hits=0 status=0 QTime=0
: >> : > > > > >> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1
: >> : > > > > r:core_node3
: >> : > > > > >> > > > > > x:images_shard1_replica_n1]
: >> : > org.apache.solr.common.SolrExc
: >> : > > > > >> eption;
: >> : > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
: >> : > > > > >> > > RemoteSolrException:
: >> : > > > > >> > > > > > Error
: >> : > > > > >> > > > > > from server at http://10.224.30.207:8983/
: >> : > > > > >> > > solr/images_shard2_replica_n4
: >> : > > > > >> > > > :
: >> : > > > > >> > > > > > Expected mime type application/octet-stream but got
: >> : > > > > >> > application/json.
: >> : > > > > >> > > > > > at
: >> : > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
: >> : > > > > >> > > > > > executeMethod(HttpSolrClient.java:607)
: >> : > > > > >> > > > > > at
: >> : > > > > >> > > > > > org.apache.solr.client.solrj.
: >> : > impl.HttpSolrClient.request(
: >> : > > > > >> > > > > > HttpSolrClient.java:255)
: >> : > > > > >> > > > > > at
: >> : > > > > >> > > > > > org.apache.solr.client.solrj.
: >> : > impl.HttpSolrClient.request(
: >> : > > > > >> > > > > > HttpSolrClient.java:244)
: >> : > > > > >> > > > > > at
: >> : > > > > >> > > > > > org.apache.solr.client.solrj.i
: >> mpl.LBHttpSolrClient.
: >> : > > > > >> > > > > > doRequest(LBHttpSolrClient.java:483)
: >> : > > > > >> > > > > > at
: >> : > > > > >> > > > > > org.apache.solr.client.solrj.
: >> : > > impl.LBHttpSolrClient.request(
: >> : > > > > >> > > > > > LBHttpSolrClient.java:413)
: >> : > > > > >> > > > > > at
: >> : > > > > >> > > > > > org.apache.solr.handler.component.
: >> : > > HttpShardHandlerFactory.
: >> : > > > > >> > > > > > makeLoadBalancedRequest(HttpSh
: >> ardHandlerFactory.java:
: >> : > 273)
: >> : > > > > >> > > > > > at
: >> : > > > > >> > > > > > org.apache.solr.handler.compon
: >> ent.HttpShardHandler.
: >> : > > > > >> > lambda$submit$0(
: >> : > > > > >> > > > > > HttpShardHandler.java:175)
: >> : > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
: >> : > > java:266)
: >> : > > > > >> > > > > > at java.util.concurrent.Executors$RunnableAdapter.
: >> : > > > > >> > > > > call(Executors.java:511)
: >> : > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
: >> : > > java:266)
: >> : > > > > >> > > > > > at
: >> : > > > > >> > > > > > com.codahale.metrics.InstrumentedExecutorService$
: >> : > > > > >> > > > > InstrumentedRunnable.run(
: >> : > > > > >> > > > > > InstrumentedExecutorService.java:176)
: >> : > > > > >> > > > > > at
: >> : > > > > >> > > > > > org.apache.solr.common.util.ExecutorUtil$
: >> : > > > > >> > MDCAwareThreadPoolExecutor.
: >> : > > > > >> > > > > > lambda$execute$0(ExecutorUtil.java:188)
: >> : > > > > >> > > > > > at
: >> : > > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
: >> : > > > > >> > > > > > ThreadPoolExecutor.java:1142)
: >> : > > > > >> > > > > > at
: >> : > > > > >> > > > > > java.util.concurrent.ThreadPoo
: >> lExecutor$Worker.run(
: >> : > > > > >> > > > > > ThreadPoolExecutor.java:617)
: >> : > > > > >> > > > > > at java.lang.Thread.run(Thread.java:745)
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1
: >> : > > > > r:core_node3
: >> : > > > > >> > > > > > x:images_shard1_replica_n1]
: >> : > org.apache.solr.core.SolrCore;
: >> : > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr
: >> path=/image
: >> : > > > > >> > params={q=id:1}
: >> : > > > > >> > > > > > status=200 QTime=17
: >> : > > > > >> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1
: >> : > > > > r:core_node3
: >> : > > > > >> > > > > > x:images_shard1_replica_n1]
: >> : > org.apache.solr.core.SolrCore;
: >> : > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr
: >> path=/image
: >> : > > > > >> > > > > > params={df=text&distrib=false&
: >> qt=/image&fl=id&fl=score&
: >> : > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
: >> : > > > > >> > > > > > http://10.224.30.207:8983/
: >> : > solr/images_shard1_replica_n1/
: >> : > > > > >> > > > > > |http://10.224.30.207:7574/
: >> : > solr/images_shard1_replica_n2/
: >> : > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
: >> : > > > 1523971232039&isShard=true&wt=
: >> : > > > > >> > javabin}
: >> : > > > > >> > > > > > hits=1 status=0 QTime=2
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > > I've implemented getcontenttype simply as
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > > @Override
: >> : > > > > >> > > > > > public String getContentType(SolrQueryRequest
: >> request,
: >> : > > > > >> > > > > > SolrQueryResponse response) {
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > > return "application/json;charset=utf-8";
: >> : > > > > >> > > > > > }
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev <
: >> : > > > mkhl@apache.org>
: >> : > > > > >> > wrote:
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > > > > Lee,
: >> : > > > > >> > > > > > > It's worth to send a stacktrace for such kind of
: >> : > > > inquiries.
: >> : > > > > >> > > > > > > I guess it goes from QueryComponent.mergeIds()
: >> or so.
: >> : > > > Shard
: >> : > > > > >> > > response
: >> : > > > > >> > > > > > should
: >> : > > > > >> > > > > > > contains <uniqueKey> from schema.xml field.
: >> : > > > > >> > > > > > > I encounter something like this while
: >> troubleshooting
: >> : > > > > >> > > > > > > https://lucene.apache.org/
: >> : > solr/guide/6_6/transforming-
: >> : > > > > >> > > > > > > result-documents.html#Transfor
: >> mingResultDocuments-
: >> : > > > > >> > > > > > > CoresandCollectionsinSolrCloud
: >> : > > > > >> > > > > > >
: >> : > > > > >> > > > > > >
: >> : > > > > >> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
: >> : > > > > >> > > > > > lee.a.carroll@googlemail.com
: >> : > > > > >> > > > > > > >
: >> : > > > > >> > > > > > > wrote:
: >> : > > > > >> > > > > > >
: >> : > > > > >> > > > > > > > I've created a custom response writer which
: >> extends
: >> : > > > > >> > > > > RawResponseWriter.
: >> : > > > > >> > > > > > > The
: >> : > > > > >> > > > > > > > basic operation is to output a single field
: >> value
: >> : > from
: >> : > > > the
: >> : > > > > >> top
: >> : > > > > >> > > > > matching
: >> : > > > > >> > > > > > > doc
: >> : > > > > >> > > > > > > > as the entire response. This works when shards
: >> = 1
: >> : > but
: >> : > > > > fails
: >> : > > > > >> > when
: >> : > > > > >> > > > > > shards
: >> : > > > > >> > > > > > > > are greater than 1.
: >> : > > > > >> > > > > > > >
: >> : > > > > >> > > > > > > > I throw an error if the field in question is
: >> missing
: >> : > > > from
: >> : > > > > >> the
: >> : > > > > >> > top
: >> : > > > > >> > > > > doc.
: >> : > > > > >> > > > > > > This
: >> : > > > > >> > > > > > > > happens when individual shards are being
: >> searched
: >> : > and
: >> : > > > only
: >> : > > > > >> id
: >> : > > > > >> > and
: >> : > > > > >> > > > > score
: >> : > > > > >> > > > > > > are
: >> : > > > > >> > > > > > > > returned. I'm sure I've committed a basic
: >> error.
: >> : > > > > >> > > > > > > >
: >> : > > > > >> > > > > > > > Lee C
: >> : > > > > >> > > > > > > >
: >> : > > > > >> > > > > > >
: >> : > > > > >> > > > > > >
: >> : > > > > >> > > > > > >
: >> : > > > > >> > > > > > > --
: >> : > > > > >> > > > > > > Sincerely yours
: >> : > > > > >> > > > > > > Mikhail Khludnev
: >> : > > > > >> > > > > > >
: >> : > > > > >> > > > > >
: >> : > > > > >> > > > >
: >> : > > > > >> > > > >
: >> : > > > > >> > > > >
: >> : > > > > >> > > > > --
: >> : > > > > >> > > > > Sincerely yours
: >> : > > > > >> > > > > Mikhail Khludnev
: >> : > > > > >> > > > >
: >> : > > > > >> > > >
: >> : > > > > >> > >
: >> : > > > > >> > >
: >> : > > > > >> > >
: >> : > > > > >> > > --
: >> : > > > > >> > > Sincerely yours
: >> : > > > > >> > > Mikhail Khludnev
: >> : > > > > >> > >
: >> : > > > > >> >
: >> : > > > > >>
: >> : > > > > >>
: >> : > > > > >>
: >> : > > > > >> --
: >> : > > > > >> Sincerely yours
: >> : > > > > >> Mikhail Khludnev
: >> : > > > > >>
: >> : > > > > >
: >> : > > > > >
: >> : > > > >
: >> : > > >
: >> : > > >
: >> : > > >
: >> : > > > --
: >> : > > > Sincerely yours
: >> : > > > Mikhail Khludnev
: >> : > > >
: >> : > >
: >> : >
: >> : >
: >> : >
: >> : > --
: >> : > Sincerely yours
: >> : > Mikhail Khludnev
: >> : >
: >> :
: >>
: >> -Hoss
: >> http://www.lucidworks.com/
: >>
: >
: >
:
-Hoss
http://www.lucidworks.com/
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Lee Carroll <le...@googlemail.com>.
Hi,
I've ended up processing the doclist in the response twice. Once in the
write method and once in getContent. Its a bit inefficient but i'm only
looking at top doc each time so probably ok.
Is their a better way to do this ?
Cheers lee C
On 22 April 2018 at 13:26, Lee Carroll <le...@googlemail.com> wrote:
> Hi,
> That works a treat. The raw response writer has a configurable base class
> which executes when no content stream is present in the response so I just
> delegate to that. I do still have an issue with writing content type on the
> http response from a value in the top document however. Although I do have
> a getContentType method which returns a value from a field in the top
> document this is called before the response writer has performed its write
> method.
>
> Whats the best way to set response headers using values from documents in
> the search result? In particular Content-Type.
>
> Cheers Lee C
>
> On 20 April 2018 at 20:03, Chris Hostetter <ho...@fucit.org>
> wrote:
>
>>
>> Invariant really means "invariant" ... nothing can change them
>>
>> In the case of "wt" this may seem weird and unhelpful, but the code that
>> handles defaults/appends/invariants is ignorant of what the params are.
>>
>> Since your writting custom code anyway, my suggestion would be that
>> perhaps you could make your custom ResponseWriter delegate to the javabin
>> responsewriter if/when you see that this is an "isShard=true" request?
>>
>>
>>
>> : Date: Thu, 19 Apr 2018 18:42:58 +0100
>> : From: Lee Carroll <le...@googlemail.com>
>> : Reply-To: solr-user@lucene.apache.org
>> : To: solr-user@lucene.apache.org
>> : Subject: Re: custom response writer which extends RawResponseWriter
>> fails when
>> : shards > 1
>> :
>> : Hi,
>> :
>> : I rewrote all of my tests to use SolrCloudTestCase rather than
>> SolrTestCaseJ4
>> : and was able to replicate the responsewriter issue and debug with a
>> sharded
>> : collection. It turned out the issue was not with my response writer
>> really
>> : but rather my config.
>> :
>> : <requestHandler name="/content" class="solr.SearchHandler">
>> : ....
>> :
>> : <lst name="invariants">
>> : <str name="wt">content</str>
>> : </lst>
>> :
>> : </requestHandler>
>> :
>> : In cloud mode having wt as an invariant breaks the collation of results
>> : from shards. Now I'm sure this is a common mistake which I've repeated
>> : (blush) but I do sort of want to actually implement my request handler
>> in
>> : this way. Is their a way to have a request handler support a single
>> : response writer but still work in cloud mode ?
>> :
>> : Could this be considered a bug ?
>> :
>> : Lee C
>> :
>> : On 18 April 2018 at 13:13, Mikhail Khludnev <mk...@apache.org> wrote:
>> :
>> : > Injecting headers might require deeper customisation up to
>> establishing own
>> : > filter or so.
>> : > Speaking regarding your own WT, there might be some issues because
>> usually
>> : > it's not a big deal to use one wt for responding user query like
>> (wt=csv)
>> : > and wt=javabin in internal communication between aggregator and
>> slaves like
>> : > it happens in wt=csv query.
>> : >
>> : > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll <
>> lee.a.carroll@googlemail.com
>> : > >
>> : > wrote:
>> : >
>> : > > Inventive. I need to control content-type of the response from the
>> : > document
>> : > > field value. I have the actual content field and the content-type
>> field
>> : > to
>> : > > use configured in the response writer. I've just noticed that the
>> xslt
>> : > > transformer allows you to do this but not controlled by document
>> values.
>> : > I
>> : > > may also need to set some headers based on content-type and perhaps
>> : > content
>> : > > size, accept ranges comes to mind. Although I might be getting
>> ahead of
>> : > > myself.
>> : > >
>> : > >
>> : > >
>> : > > On 18 April 2018 at 12:05, Mikhail Khludnev <mk...@apache.org>
>> wrote:
>> : > >
>> : > > > well ..
>> : > > > what if
>> : > > > http://localhost:8983/solr/images/select?fl=content&q=id:
>> : > > 1&start=1&wt=csv&
>> : > > > csv.separator=&csv.encapsulator&csv.null=null
>> : > > > ?
>> : > > >
>> : > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll <
>> : > > lee.a.carroll@googlemail.com
>> : > > > >
>> : > > > wrote:
>> : > > >
>> : > > > > sorry cut n paste error i'd get
>> : > > > >
>> : > > > > {
>> : > > > > "responseHeader":{
>> : > > > > "zkConnected":true,
>> : > > > > "status":0,
>> : > > > > "QTime":0,
>> : > > > > "params":{
>> : > > > > "q":"*:*",
>> : > > > > "fl":"content",
>> : > > > > "rows":"1"}},
>> : > > > > "response":{"numFound":1,"start":0,"docs":[
>> : > > > > {
>> : > > > > "content":"my-content-value"}]
>> : > > > > }}
>> : > > > >
>> : > > > >
>> : > > > > but you get my point
>> : > > > >
>> : > > > >
>> : > > > >
>> : > > > > On 18 April 2018 at 11:13, Lee Carroll <
>> lee.a.carroll@googlemail.com
>> : > >
>> : > > > > wrote:
>> : > > > >
>> : > > > > > for http://localhost:8983/solr/images/select?fl=content&q=id:
>> : > > 1&start=1
>> : > > > > >
>> : > > > > > I'd get
>> : > > > > >
>> : > > > > > {
>> : > > > > > "responseHeader":{
>> : > > > > > "zkConnected":true,
>> : > > > > > "status":0,
>> : > > > > > "QTime":1,
>> : > > > > > "params":{
>> : > > > > > "q":"*:*",
>> : > > > > > "_":"1524046333220"}},
>> : > > > > > "response":{"numFound":1,"start":0,"docs":[
>> : > > > > > {
>> : > > > > > "id":"1",
>> : > > > > > "content":"my-content-value",
>> : > > > > > "*content-type*":"text/plain"}]
>> : > > > > > }}
>> : > > > > >
>> : > > > > > when i want
>> : > > > > >
>> : > > > > > my-content-value
>> : > > > > >
>> : > > > > >
>> : > > > > >
>> : > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev <mk...@apache.org>
>> : > wrote:
>> : > > > > >
>> : > > > > >> Lee, from this description I don see why it can't be
>> addressed by
>> : > > > > fl,rows
>> : > > > > >> params. What makes it different form the typical Solr usage?
>> : > > > > >>
>> : > > > > >>
>> : > > > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll <
>> : > > > > >> lee.a.carroll@googlemail.com>
>> : > > > > >> wrote:
>> : > > > > >>
>> : > > > > >> > Sure, we want to return a single field's value for the top
>> : > > matching
>> : > > > > >> > document for a given query. Bare content rather than a full
>> : > search
>> : > > > > >> result
>> : > > > > >> > listing.
>> : > > > > >> >
>> : > > > > >> > To be concrete:
>> : > > > > >> >
>> : > > > > >> > For a schema of fields id [unique key],
>> : > > > content[stored],content-type[
>> : > > > > >> > stored]
>> : > > > > >> > For a request:
>> : > > > > >> >
>> : > > > > >> > 1. Request URL:
>> : > > > > >> > https://localhost/solr/content?q=id:1
>> : > > > > >> > 2. Request Method:
>> : > > > > >> > GET
>> : > > > > >> >
>> : > > > > >> > We get a response
>> : > > > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type:
>> : > [content-type
>> : > > > > value]
>> : > > > > >> >
>> : > > > > >> > and the body to be the raw value of content
>> : > > > > >> >
>> : > > > > >> > In short clients consume directly the most relevant
>> "content"
>> : > > > returned
>> : > > > > >> from
>> : > > > > >> > solr queries they construct.
>> : > > > > >> >
>> : > > > > >> > Naively I've implemented a subclass of RawResponseWriter
>> which
>> : > > takes
>> : > > > > the
>> : > > > > >> > first docs values and adds them to the appended "content"
>> : > stream.
>> : > > > > >> Should I
>> : > > > > >> > selectively add the content stream depending on if this is
>> the
>> : > > final
>> : > > > > >> > aggregation of cloud results (and provide a base class
>> writer to
>> : > > act
>> : > > > > if
>> : > > > > >> > not), if so how do I know its the final aggregation. Or is
>> : > adding
>> : > > > the
>> : > > > > >> > content stream within the response writer a bad idea.
>> Should
>> : > that
>> : > > be
>> : > > > > >> being
>> : > > > > >> > added to the response somewhere else?
>> : > > > > >> >
>> : > > > > >> > Failing all of the above is asking about response writer
>> an X /
>> : > Y
>> : > > > > >> problem.
>> : > > > > >> > Is their a better way to achieve the above. I'd looked at
>> : > > > transforming
>> : > > > > >> > response xml but that seemed not to offer a complete bare
>> slate.
>> : > > > > >> >
>> : > > > > >> > Cheers Lee C
>> : > > > > >> >
>> : > > > > >> >
>> : > > > > >> > On 17 April 2018 at 21:36, Mikhail Khludnev <
>> mkhl@apache.org>
>> : > > > wrote:
>> : > > > > >> >
>> : > > > > >> > > In distributed search response writer is used twice
>> : > > > > >> > > https://lucene.apache.org/solr/guide/7_1/distributed-
>> : > > > requests.html
>> : > > > > >> > > once slave node that's where response writer yields
>> "json"
>> : > > content
>> : > > > > >> and it
>> : > > > > >> > > upset aggregator node which is expect only javabin.
>> : > > > > >> > > I hardly can comment on rrw, it's probably used for
>> responding
>> : > > > > >> separate
>> : > > > > >> > > files in distrib=false mode.
>> : > > > > >> > > You can start from describing why you need to create own
>> : > > response
>> : > > > > >> writer.
>> : > > > > >> > >
>> : > > > > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <
>> : > > > > >> > lee.a.carroll@googlemail.com
>> : > > > > >> > > >
>> : > > > > >> > > wrote:
>> : > > > > >> > >
>> : > > > > >> > > > Ok. My expectation was the response writer would not
>> be used
>> : > > > until
>> : > > > > >> the
>> : > > > > >> > > > final serialization of the result. If my response
>> writer
>> : > > breaks
>> : > > > > the
>> : > > > > >> > > > response writer contract, exactly the way
>> rawResponseWriter
>> : > > does
>> : > > > > and
>> : > > > > >> > just
>> : > > > > >> > > > out puts a filed value how does that work? Does
>> : > > > rawResponseWriter
>> : > > > > >> > support
>> : > > > > >> > > > cloud mode?
>> : > > > > >> > > >
>> : > > > > >> > > >
>> : > > > > >> > > >
>> : > > > > >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev <
>> : > mkhl@apache.org>
>> : > > > > >> wrote:
>> : > > > > >> > > >
>> : > > > > >> > > > > That's what should happen.
>> : > > > > >> > > > >
>> : > > > > >> > > > > Expected mime type application/octet-stream but got
>> : > > > > >> application/json.
>> : > > > > >> > > > >
>> : > > > > >> > > > > Distributed search coordinator expect to merge slave
>> : > > responses
>> : > > > > in
>> : > > > > >> > > javabin
>> : > > > > >> > > > > format. But slave's wt indicated json.
>> : > > > > >> > > > > As far as I know only javabin might be used to
>> distributed
>> : > > > > search
>> : > > > > >> > > > > underneath. Coordinator itself might yield json.
>> : > > > > >> > > > >
>> : > > > > >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
>> : > > > > >> > > > lee.a.carroll@googlemail.com
>> : > > > > >> > > > > >
>> : > > > > >> > > > > wrote:
>> : > > > > >> > > > >
>> : > > > > >> > > > > > Sure
>> : > > > > >> > > > > >
>> : > > > > >> > > > > > with 1 shard 1 replica this request works fine
>> : > > > > >> > > > > >
>> : > > > > >> > > > > > 1. Request URL:
>> : > > > > >> > > > > > http://localhost:8983/solr/images/image?q=id:1
>> : > > > > >> > > > > > 2. Request Method:
>> : > > > > >> > > > > > GET
>> : > > > > >> > > > > > 3. Status Code:
>> : > > > > >> > > > > > 200 OK
>> : > > > > >> > > > > >
>> : > > > > >> > > > > > logs are clean
>> : > > > > >> > > > > >
>> : > > > > >> > > > > > with 2 shards 2 replicas the same request fails
>> and in
>> : > the
>> : > > > > logs
>> : > > > > >> > > > > >
>> : > > > > >> > > > > >
>> : > > > > >> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2
>> : > > > > r:core_node7
>> : > > > > >> > > > > > x:images_shard2_replica_n4]
>> : > org.apache.solr.core.SolrCore;
>> : > > > > >> > > > > > [images_shard2_replica_n4] webapp=/solr
>> path=/image
>> : > > > > >> > > > > > params={df=text&distrib=false&
>> qt=/image&fl=id&fl=score&
>> : > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
>> : > > > > >> > > > > > http://10.224.30.207:8983/
>> : > solr/images_shard2_replica_n4/
>> : > > > > >> > > > > > |http://10.224.30.207:7574/
>> : > solr/images_shard2_replica_n6/
>> : > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
>> : > > > 1523971232039&isShard=true&wt=
>> : > > > > >> > javabin}
>> : > > > > >> > > > > > hits=0 status=0 QTime=0
>> : > > > > >> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1
>> : > > > > r:core_node3
>> : > > > > >> > > > > > x:images_shard1_replica_n1]
>> : > org.apache.solr.common.SolrExc
>> : > > > > >> eption;
>> : > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
>> : > > > > >> > > RemoteSolrException:
>> : > > > > >> > > > > > Error
>> : > > > > >> > > > > > from server at http://10.224.30.207:8983/
>> : > > > > >> > > solr/images_shard2_replica_n4
>> : > > > > >> > > > :
>> : > > > > >> > > > > > Expected mime type application/octet-stream but got
>> : > > > > >> > application/json.
>> : > > > > >> > > > > > at
>> : > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
>> : > > > > >> > > > > > executeMethod(HttpSolrClient.java:607)
>> : > > > > >> > > > > > at
>> : > > > > >> > > > > > org.apache.solr.client.solrj.
>> : > impl.HttpSolrClient.request(
>> : > > > > >> > > > > > HttpSolrClient.java:255)
>> : > > > > >> > > > > > at
>> : > > > > >> > > > > > org.apache.solr.client.solrj.
>> : > impl.HttpSolrClient.request(
>> : > > > > >> > > > > > HttpSolrClient.java:244)
>> : > > > > >> > > > > > at
>> : > > > > >> > > > > > org.apache.solr.client.solrj.i
>> mpl.LBHttpSolrClient.
>> : > > > > >> > > > > > doRequest(LBHttpSolrClient.java:483)
>> : > > > > >> > > > > > at
>> : > > > > >> > > > > > org.apache.solr.client.solrj.
>> : > > impl.LBHttpSolrClient.request(
>> : > > > > >> > > > > > LBHttpSolrClient.java:413)
>> : > > > > >> > > > > > at
>> : > > > > >> > > > > > org.apache.solr.handler.component.
>> : > > HttpShardHandlerFactory.
>> : > > > > >> > > > > > makeLoadBalancedRequest(HttpSh
>> ardHandlerFactory.java:
>> : > 273)
>> : > > > > >> > > > > > at
>> : > > > > >> > > > > > org.apache.solr.handler.compon
>> ent.HttpShardHandler.
>> : > > > > >> > lambda$submit$0(
>> : > > > > >> > > > > > HttpShardHandler.java:175)
>> : > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
>> : > > java:266)
>> : > > > > >> > > > > > at java.util.concurrent.Executors$RunnableAdapter.
>> : > > > > >> > > > > call(Executors.java:511)
>> : > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
>> : > > java:266)
>> : > > > > >> > > > > > at
>> : > > > > >> > > > > > com.codahale.metrics.InstrumentedExecutorService$
>> : > > > > >> > > > > InstrumentedRunnable.run(
>> : > > > > >> > > > > > InstrumentedExecutorService.java:176)
>> : > > > > >> > > > > > at
>> : > > > > >> > > > > > org.apache.solr.common.util.ExecutorUtil$
>> : > > > > >> > MDCAwareThreadPoolExecutor.
>> : > > > > >> > > > > > lambda$execute$0(ExecutorUtil.java:188)
>> : > > > > >> > > > > > at
>> : > > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
>> : > > > > >> > > > > > ThreadPoolExecutor.java:1142)
>> : > > > > >> > > > > > at
>> : > > > > >> > > > > > java.util.concurrent.ThreadPoo
>> lExecutor$Worker.run(
>> : > > > > >> > > > > > ThreadPoolExecutor.java:617)
>> : > > > > >> > > > > > at java.lang.Thread.run(Thread.java:745)
>> : > > > > >> > > > > >
>> : > > > > >> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1
>> : > > > > r:core_node3
>> : > > > > >> > > > > > x:images_shard1_replica_n1]
>> : > org.apache.solr.core.SolrCore;
>> : > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr
>> path=/image
>> : > > > > >> > params={q=id:1}
>> : > > > > >> > > > > > status=200 QTime=17
>> : > > > > >> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1
>> : > > > > r:core_node3
>> : > > > > >> > > > > > x:images_shard1_replica_n1]
>> : > org.apache.solr.core.SolrCore;
>> : > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr
>> path=/image
>> : > > > > >> > > > > > params={df=text&distrib=false&
>> qt=/image&fl=id&fl=score&
>> : > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
>> : > > > > >> > > > > > http://10.224.30.207:8983/
>> : > solr/images_shard1_replica_n1/
>> : > > > > >> > > > > > |http://10.224.30.207:7574/
>> : > solr/images_shard1_replica_n2/
>> : > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
>> : > > > 1523971232039&isShard=true&wt=
>> : > > > > >> > javabin}
>> : > > > > >> > > > > > hits=1 status=0 QTime=2
>> : > > > > >> > > > > >
>> : > > > > >> > > > > >
>> : > > > > >> > > > > > I've implemented getcontenttype simply as
>> : > > > > >> > > > > >
>> : > > > > >> > > > > > @Override
>> : > > > > >> > > > > > public String getContentType(SolrQueryRequest
>> request,
>> : > > > > >> > > > > > SolrQueryResponse response) {
>> : > > > > >> > > > > >
>> : > > > > >> > > > > > return "application/json;charset=utf-8";
>> : > > > > >> > > > > > }
>> : > > > > >> > > > > >
>> : > > > > >> > > > > >
>> : > > > > >> > > > > >
>> : > > > > >> > > > > >
>> : > > > > >> > > > > >
>> : > > > > >> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev <
>> : > > > mkhl@apache.org>
>> : > > > > >> > wrote:
>> : > > > > >> > > > > >
>> : > > > > >> > > > > > > Lee,
>> : > > > > >> > > > > > > It's worth to send a stacktrace for such kind of
>> : > > > inquiries.
>> : > > > > >> > > > > > > I guess it goes from QueryComponent.mergeIds()
>> or so.
>> : > > > Shard
>> : > > > > >> > > response
>> : > > > > >> > > > > > should
>> : > > > > >> > > > > > > contains <uniqueKey> from schema.xml field.
>> : > > > > >> > > > > > > I encounter something like this while
>> troubleshooting
>> : > > > > >> > > > > > > https://lucene.apache.org/
>> : > solr/guide/6_6/transforming-
>> : > > > > >> > > > > > > result-documents.html#Transfor
>> mingResultDocuments-
>> : > > > > >> > > > > > > CoresandCollectionsinSolrCloud
>> : > > > > >> > > > > > >
>> : > > > > >> > > > > > >
>> : > > > > >> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
>> : > > > > >> > > > > > lee.a.carroll@googlemail.com
>> : > > > > >> > > > > > > >
>> : > > > > >> > > > > > > wrote:
>> : > > > > >> > > > > > >
>> : > > > > >> > > > > > > > I've created a custom response writer which
>> extends
>> : > > > > >> > > > > RawResponseWriter.
>> : > > > > >> > > > > > > The
>> : > > > > >> > > > > > > > basic operation is to output a single field
>> value
>> : > from
>> : > > > the
>> : > > > > >> top
>> : > > > > >> > > > > matching
>> : > > > > >> > > > > > > doc
>> : > > > > >> > > > > > > > as the entire response. This works when shards
>> = 1
>> : > but
>> : > > > > fails
>> : > > > > >> > when
>> : > > > > >> > > > > > shards
>> : > > > > >> > > > > > > > are greater than 1.
>> : > > > > >> > > > > > > >
>> : > > > > >> > > > > > > > I throw an error if the field in question is
>> missing
>> : > > > from
>> : > > > > >> the
>> : > > > > >> > top
>> : > > > > >> > > > > doc.
>> : > > > > >> > > > > > > This
>> : > > > > >> > > > > > > > happens when individual shards are being
>> searched
>> : > and
>> : > > > only
>> : > > > > >> id
>> : > > > > >> > and
>> : > > > > >> > > > > score
>> : > > > > >> > > > > > > are
>> : > > > > >> > > > > > > > returned. I'm sure I've committed a basic
>> error.
>> : > > > > >> > > > > > > >
>> : > > > > >> > > > > > > > Lee C
>> : > > > > >> > > > > > > >
>> : > > > > >> > > > > > >
>> : > > > > >> > > > > > >
>> : > > > > >> > > > > > >
>> : > > > > >> > > > > > > --
>> : > > > > >> > > > > > > Sincerely yours
>> : > > > > >> > > > > > > Mikhail Khludnev
>> : > > > > >> > > > > > >
>> : > > > > >> > > > > >
>> : > > > > >> > > > >
>> : > > > > >> > > > >
>> : > > > > >> > > > >
>> : > > > > >> > > > > --
>> : > > > > >> > > > > Sincerely yours
>> : > > > > >> > > > > Mikhail Khludnev
>> : > > > > >> > > > >
>> : > > > > >> > > >
>> : > > > > >> > >
>> : > > > > >> > >
>> : > > > > >> > >
>> : > > > > >> > > --
>> : > > > > >> > > Sincerely yours
>> : > > > > >> > > Mikhail Khludnev
>> : > > > > >> > >
>> : > > > > >> >
>> : > > > > >>
>> : > > > > >>
>> : > > > > >>
>> : > > > > >> --
>> : > > > > >> Sincerely yours
>> : > > > > >> Mikhail Khludnev
>> : > > > > >>
>> : > > > > >
>> : > > > > >
>> : > > > >
>> : > > >
>> : > > >
>> : > > >
>> : > > > --
>> : > > > Sincerely yours
>> : > > > Mikhail Khludnev
>> : > > >
>> : > >
>> : >
>> : >
>> : >
>> : > --
>> : > Sincerely yours
>> : > Mikhail Khludnev
>> : >
>> :
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>
>
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Lee Carroll <le...@googlemail.com>.
Hi,
That works a treat. The raw response writer has a configurable base class
which executes when no content stream is present in the response so I just
delegate to that. I do still have an issue with writing content type on the
http response from a value in the top document however. Although I do have
a getContentType method which returns a value from a field in the top
document this is called before the response writer has performed its write
method.
Whats the best way to set response headers using values from documents in
the search result? In particular Content-Type.
Cheers Lee C
On 20 April 2018 at 20:03, Chris Hostetter <ho...@fucit.org> wrote:
>
> Invariant really means "invariant" ... nothing can change them
>
> In the case of "wt" this may seem weird and unhelpful, but the code that
> handles defaults/appends/invariants is ignorant of what the params are.
>
> Since your writting custom code anyway, my suggestion would be that
> perhaps you could make your custom ResponseWriter delegate to the javabin
> responsewriter if/when you see that this is an "isShard=true" request?
>
>
>
> : Date: Thu, 19 Apr 2018 18:42:58 +0100
> : From: Lee Carroll <le...@googlemail.com>
> : Reply-To: solr-user@lucene.apache.org
> : To: solr-user@lucene.apache.org
> : Subject: Re: custom response writer which extends RawResponseWriter
> fails when
> : shards > 1
> :
> : Hi,
> :
> : I rewrote all of my tests to use SolrCloudTestCase rather than
> SolrTestCaseJ4
> : and was able to replicate the responsewriter issue and debug with a
> sharded
> : collection. It turned out the issue was not with my response writer
> really
> : but rather my config.
> :
> : <requestHandler name="/content" class="solr.SearchHandler">
> : ....
> :
> : <lst name="invariants">
> : <str name="wt">content</str>
> : </lst>
> :
> : </requestHandler>
> :
> : In cloud mode having wt as an invariant breaks the collation of results
> : from shards. Now I'm sure this is a common mistake which I've repeated
> : (blush) but I do sort of want to actually implement my request handler in
> : this way. Is their a way to have a request handler support a single
> : response writer but still work in cloud mode ?
> :
> : Could this be considered a bug ?
> :
> : Lee C
> :
> : On 18 April 2018 at 13:13, Mikhail Khludnev <mk...@apache.org> wrote:
> :
> : > Injecting headers might require deeper customisation up to
> establishing own
> : > filter or so.
> : > Speaking regarding your own WT, there might be some issues because
> usually
> : > it's not a big deal to use one wt for responding user query like
> (wt=csv)
> : > and wt=javabin in internal communication between aggregator and slaves
> like
> : > it happens in wt=csv query.
> : >
> : > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll <
> lee.a.carroll@googlemail.com
> : > >
> : > wrote:
> : >
> : > > Inventive. I need to control content-type of the response from the
> : > document
> : > > field value. I have the actual content field and the content-type
> field
> : > to
> : > > use configured in the response writer. I've just noticed that the
> xslt
> : > > transformer allows you to do this but not controlled by document
> values.
> : > I
> : > > may also need to set some headers based on content-type and perhaps
> : > content
> : > > size, accept ranges comes to mind. Although I might be getting ahead
> of
> : > > myself.
> : > >
> : > >
> : > >
> : > > On 18 April 2018 at 12:05, Mikhail Khludnev <mk...@apache.org> wrote:
> : > >
> : > > > well ..
> : > > > what if
> : > > > http://localhost:8983/solr/images/select?fl=content&q=id:
> : > > 1&start=1&wt=csv&
> : > > > csv.separator=&csv.encapsulator&csv.null=null
> : > > > ?
> : > > >
> : > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll <
> : > > lee.a.carroll@googlemail.com
> : > > > >
> : > > > wrote:
> : > > >
> : > > > > sorry cut n paste error i'd get
> : > > > >
> : > > > > {
> : > > > > "responseHeader":{
> : > > > > "zkConnected":true,
> : > > > > "status":0,
> : > > > > "QTime":0,
> : > > > > "params":{
> : > > > > "q":"*:*",
> : > > > > "fl":"content",
> : > > > > "rows":"1"}},
> : > > > > "response":{"numFound":1,"start":0,"docs":[
> : > > > > {
> : > > > > "content":"my-content-value"}]
> : > > > > }}
> : > > > >
> : > > > >
> : > > > > but you get my point
> : > > > >
> : > > > >
> : > > > >
> : > > > > On 18 April 2018 at 11:13, Lee Carroll <
> lee.a.carroll@googlemail.com
> : > >
> : > > > > wrote:
> : > > > >
> : > > > > > for http://localhost:8983/solr/images/select?fl=content&q=id:
> : > > 1&start=1
> : > > > > >
> : > > > > > I'd get
> : > > > > >
> : > > > > > {
> : > > > > > "responseHeader":{
> : > > > > > "zkConnected":true,
> : > > > > > "status":0,
> : > > > > > "QTime":1,
> : > > > > > "params":{
> : > > > > > "q":"*:*",
> : > > > > > "_":"1524046333220"}},
> : > > > > > "response":{"numFound":1,"start":0,"docs":[
> : > > > > > {
> : > > > > > "id":"1",
> : > > > > > "content":"my-content-value",
> : > > > > > "*content-type*":"text/plain"}]
> : > > > > > }}
> : > > > > >
> : > > > > > when i want
> : > > > > >
> : > > > > > my-content-value
> : > > > > >
> : > > > > >
> : > > > > >
> : > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev <mk...@apache.org>
> : > wrote:
> : > > > > >
> : > > > > >> Lee, from this description I don see why it can't be
> addressed by
> : > > > > fl,rows
> : > > > > >> params. What makes it different form the typical Solr usage?
> : > > > > >>
> : > > > > >>
> : > > > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll <
> : > > > > >> lee.a.carroll@googlemail.com>
> : > > > > >> wrote:
> : > > > > >>
> : > > > > >> > Sure, we want to return a single field's value for the top
> : > > matching
> : > > > > >> > document for a given query. Bare content rather than a full
> : > search
> : > > > > >> result
> : > > > > >> > listing.
> : > > > > >> >
> : > > > > >> > To be concrete:
> : > > > > >> >
> : > > > > >> > For a schema of fields id [unique key],
> : > > > content[stored],content-type[
> : > > > > >> > stored]
> : > > > > >> > For a request:
> : > > > > >> >
> : > > > > >> > 1. Request URL:
> : > > > > >> > https://localhost/solr/content?q=id:1
> : > > > > >> > 2. Request Method:
> : > > > > >> > GET
> : > > > > >> >
> : > > > > >> > We get a response
> : > > > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type:
> : > [content-type
> : > > > > value]
> : > > > > >> >
> : > > > > >> > and the body to be the raw value of content
> : > > > > >> >
> : > > > > >> > In short clients consume directly the most relevant
> "content"
> : > > > returned
> : > > > > >> from
> : > > > > >> > solr queries they construct.
> : > > > > >> >
> : > > > > >> > Naively I've implemented a subclass of RawResponseWriter
> which
> : > > takes
> : > > > > the
> : > > > > >> > first docs values and adds them to the appended "content"
> : > stream.
> : > > > > >> Should I
> : > > > > >> > selectively add the content stream depending on if this is
> the
> : > > final
> : > > > > >> > aggregation of cloud results (and provide a base class
> writer to
> : > > act
> : > > > > if
> : > > > > >> > not), if so how do I know its the final aggregation. Or is
> : > adding
> : > > > the
> : > > > > >> > content stream within the response writer a bad idea. Should
> : > that
> : > > be
> : > > > > >> being
> : > > > > >> > added to the response somewhere else?
> : > > > > >> >
> : > > > > >> > Failing all of the above is asking about response writer an
> X /
> : > Y
> : > > > > >> problem.
> : > > > > >> > Is their a better way to achieve the above. I'd looked at
> : > > > transforming
> : > > > > >> > response xml but that seemed not to offer a complete bare
> slate.
> : > > > > >> >
> : > > > > >> > Cheers Lee C
> : > > > > >> >
> : > > > > >> >
> : > > > > >> > On 17 April 2018 at 21:36, Mikhail Khludnev <
> mkhl@apache.org>
> : > > > wrote:
> : > > > > >> >
> : > > > > >> > > In distributed search response writer is used twice
> : > > > > >> > > https://lucene.apache.org/solr/guide/7_1/distributed-
> : > > > requests.html
> : > > > > >> > > once slave node that's where response writer yields "json"
> : > > content
> : > > > > >> and it
> : > > > > >> > > upset aggregator node which is expect only javabin.
> : > > > > >> > > I hardly can comment on rrw, it's probably used for
> responding
> : > > > > >> separate
> : > > > > >> > > files in distrib=false mode.
> : > > > > >> > > You can start from describing why you need to create own
> : > > response
> : > > > > >> writer.
> : > > > > >> > >
> : > > > > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <
> : > > > > >> > lee.a.carroll@googlemail.com
> : > > > > >> > > >
> : > > > > >> > > wrote:
> : > > > > >> > >
> : > > > > >> > > > Ok. My expectation was the response writer would not be
> used
> : > > > until
> : > > > > >> the
> : > > > > >> > > > final serialization of the result. If my response writer
> : > > breaks
> : > > > > the
> : > > > > >> > > > response writer contract, exactly the way
> rawResponseWriter
> : > > does
> : > > > > and
> : > > > > >> > just
> : > > > > >> > > > out puts a filed value how does that work? Does
> : > > > rawResponseWriter
> : > > > > >> > support
> : > > > > >> > > > cloud mode?
> : > > > > >> > > >
> : > > > > >> > > >
> : > > > > >> > > >
> : > > > > >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev <
> : > mkhl@apache.org>
> : > > > > >> wrote:
> : > > > > >> > > >
> : > > > > >> > > > > That's what should happen.
> : > > > > >> > > > >
> : > > > > >> > > > > Expected mime type application/octet-stream but got
> : > > > > >> application/json.
> : > > > > >> > > > >
> : > > > > >> > > > > Distributed search coordinator expect to merge slave
> : > > responses
> : > > > > in
> : > > > > >> > > javabin
> : > > > > >> > > > > format. But slave's wt indicated json.
> : > > > > >> > > > > As far as I know only javabin might be used to
> distributed
> : > > > > search
> : > > > > >> > > > > underneath. Coordinator itself might yield json.
> : > > > > >> > > > >
> : > > > > >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
> : > > > > >> > > > lee.a.carroll@googlemail.com
> : > > > > >> > > > > >
> : > > > > >> > > > > wrote:
> : > > > > >> > > > >
> : > > > > >> > > > > > Sure
> : > > > > >> > > > > >
> : > > > > >> > > > > > with 1 shard 1 replica this request works fine
> : > > > > >> > > > > >
> : > > > > >> > > > > > 1. Request URL:
> : > > > > >> > > > > > http://localhost:8983/solr/images/image?q=id:1
> : > > > > >> > > > > > 2. Request Method:
> : > > > > >> > > > > > GET
> : > > > > >> > > > > > 3. Status Code:
> : > > > > >> > > > > > 200 OK
> : > > > > >> > > > > >
> : > > > > >> > > > > > logs are clean
> : > > > > >> > > > > >
> : > > > > >> > > > > > with 2 shards 2 replicas the same request fails and
> in
> : > the
> : > > > > logs
> : > > > > >> > > > > >
> : > > > > >> > > > > >
> : > > > > >> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2
> : > > > > r:core_node7
> : > > > > >> > > > > > x:images_shard2_replica_n4]
> : > org.apache.solr.core.SolrCore;
> : > > > > >> > > > > > [images_shard2_replica_n4] webapp=/solr path=/image
> : > > > > >> > > > > > params={df=text&distrib=false&
> qt=/image&fl=id&fl=score&
> : > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> : > > > > >> > > > > > http://10.224.30.207:8983/
> : > solr/images_shard2_replica_n4/
> : > > > > >> > > > > > |http://10.224.30.207:7574/
> : > solr/images_shard2_replica_n6/
> : > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
> : > > > 1523971232039&isShard=true&wt=
> : > > > > >> > javabin}
> : > > > > >> > > > > > hits=0 status=0 QTime=0
> : > > > > >> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1
> : > > > > r:core_node3
> : > > > > >> > > > > > x:images_shard1_replica_n1]
> : > org.apache.solr.common.SolrExc
> : > > > > >> eption;
> : > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
> : > > > > >> > > RemoteSolrException:
> : > > > > >> > > > > > Error
> : > > > > >> > > > > > from server at http://10.224.30.207:8983/
> : > > > > >> > > solr/images_shard2_replica_n4
> : > > > > >> > > > :
> : > > > > >> > > > > > Expected mime type application/octet-stream but got
> : > > > > >> > application/json.
> : > > > > >> > > > > > at
> : > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
> : > > > > >> > > > > > executeMethod(HttpSolrClient.java:607)
> : > > > > >> > > > > > at
> : > > > > >> > > > > > org.apache.solr.client.solrj.
> : > impl.HttpSolrClient.request(
> : > > > > >> > > > > > HttpSolrClient.java:255)
> : > > > > >> > > > > > at
> : > > > > >> > > > > > org.apache.solr.client.solrj.
> : > impl.HttpSolrClient.request(
> : > > > > >> > > > > > HttpSolrClient.java:244)
> : > > > > >> > > > > > at
> : > > > > >> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> : > > > > >> > > > > > doRequest(LBHttpSolrClient.java:483)
> : > > > > >> > > > > > at
> : > > > > >> > > > > > org.apache.solr.client.solrj.
> : > > impl.LBHttpSolrClient.request(
> : > > > > >> > > > > > LBHttpSolrClient.java:413)
> : > > > > >> > > > > > at
> : > > > > >> > > > > > org.apache.solr.handler.component.
> : > > HttpShardHandlerFactory.
> : > > > > >> > > > > > makeLoadBalancedRequest(
> HttpShardHandlerFactory.java:
> : > 273)
> : > > > > >> > > > > > at
> : > > > > >> > > > > > org.apache.solr.handler.component.HttpShardHandler.
> : > > > > >> > lambda$submit$0(
> : > > > > >> > > > > > HttpShardHandler.java:175)
> : > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
> : > > java:266)
> : > > > > >> > > > > > at java.util.concurrent.Executors$RunnableAdapter.
> : > > > > >> > > > > call(Executors.java:511)
> : > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
> : > > java:266)
> : > > > > >> > > > > > at
> : > > > > >> > > > > > com.codahale.metrics.InstrumentedExecutorService$
> : > > > > >> > > > > InstrumentedRunnable.run(
> : > > > > >> > > > > > InstrumentedExecutorService.java:176)
> : > > > > >> > > > > > at
> : > > > > >> > > > > > org.apache.solr.common.util.ExecutorUtil$
> : > > > > >> > MDCAwareThreadPoolExecutor.
> : > > > > >> > > > > > lambda$execute$0(ExecutorUtil.java:188)
> : > > > > >> > > > > > at
> : > > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> : > > > > >> > > > > > ThreadPoolExecutor.java:1142)
> : > > > > >> > > > > > at
> : > > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> : > > > > >> > > > > > ThreadPoolExecutor.java:617)
> : > > > > >> > > > > > at java.lang.Thread.run(Thread.java:745)
> : > > > > >> > > > > >
> : > > > > >> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1
> : > > > > r:core_node3
> : > > > > >> > > > > > x:images_shard1_replica_n1]
> : > org.apache.solr.core.SolrCore;
> : > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> : > > > > >> > params={q=id:1}
> : > > > > >> > > > > > status=200 QTime=17
> : > > > > >> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1
> : > > > > r:core_node3
> : > > > > >> > > > > > x:images_shard1_replica_n1]
> : > org.apache.solr.core.SolrCore;
> : > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> : > > > > >> > > > > > params={df=text&distrib=false&
> qt=/image&fl=id&fl=score&
> : > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> : > > > > >> > > > > > http://10.224.30.207:8983/
> : > solr/images_shard1_replica_n1/
> : > > > > >> > > > > > |http://10.224.30.207:7574/
> : > solr/images_shard1_replica_n2/
> : > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
> : > > > 1523971232039&isShard=true&wt=
> : > > > > >> > javabin}
> : > > > > >> > > > > > hits=1 status=0 QTime=2
> : > > > > >> > > > > >
> : > > > > >> > > > > >
> : > > > > >> > > > > > I've implemented getcontenttype simply as
> : > > > > >> > > > > >
> : > > > > >> > > > > > @Override
> : > > > > >> > > > > > public String getContentType(SolrQueryRequest
> request,
> : > > > > >> > > > > > SolrQueryResponse response) {
> : > > > > >> > > > > >
> : > > > > >> > > > > > return "application/json;charset=utf-8";
> : > > > > >> > > > > > }
> : > > > > >> > > > > >
> : > > > > >> > > > > >
> : > > > > >> > > > > >
> : > > > > >> > > > > >
> : > > > > >> > > > > >
> : > > > > >> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev <
> : > > > mkhl@apache.org>
> : > > > > >> > wrote:
> : > > > > >> > > > > >
> : > > > > >> > > > > > > Lee,
> : > > > > >> > > > > > > It's worth to send a stacktrace for such kind of
> : > > > inquiries.
> : > > > > >> > > > > > > I guess it goes from QueryComponent.mergeIds() or
> so.
> : > > > Shard
> : > > > > >> > > response
> : > > > > >> > > > > > should
> : > > > > >> > > > > > > contains <uniqueKey> from schema.xml field.
> : > > > > >> > > > > > > I encounter something like this while
> troubleshooting
> : > > > > >> > > > > > > https://lucene.apache.org/
> : > solr/guide/6_6/transforming-
> : > > > > >> > > > > > > result-documents.html#
> TransformingResultDocuments-
> : > > > > >> > > > > > > CoresandCollectionsinSolrCloud
> : > > > > >> > > > > > >
> : > > > > >> > > > > > >
> : > > > > >> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
> : > > > > >> > > > > > lee.a.carroll@googlemail.com
> : > > > > >> > > > > > > >
> : > > > > >> > > > > > > wrote:
> : > > > > >> > > > > > >
> : > > > > >> > > > > > > > I've created a custom response writer which
> extends
> : > > > > >> > > > > RawResponseWriter.
> : > > > > >> > > > > > > The
> : > > > > >> > > > > > > > basic operation is to output a single field
> value
> : > from
> : > > > the
> : > > > > >> top
> : > > > > >> > > > > matching
> : > > > > >> > > > > > > doc
> : > > > > >> > > > > > > > as the entire response. This works when shards
> = 1
> : > but
> : > > > > fails
> : > > > > >> > when
> : > > > > >> > > > > > shards
> : > > > > >> > > > > > > > are greater than 1.
> : > > > > >> > > > > > > >
> : > > > > >> > > > > > > > I throw an error if the field in question is
> missing
> : > > > from
> : > > > > >> the
> : > > > > >> > top
> : > > > > >> > > > > doc.
> : > > > > >> > > > > > > This
> : > > > > >> > > > > > > > happens when individual shards are being
> searched
> : > and
> : > > > only
> : > > > > >> id
> : > > > > >> > and
> : > > > > >> > > > > score
> : > > > > >> > > > > > > are
> : > > > > >> > > > > > > > returned. I'm sure I've committed a basic error.
> : > > > > >> > > > > > > >
> : > > > > >> > > > > > > > Lee C
> : > > > > >> > > > > > > >
> : > > > > >> > > > > > >
> : > > > > >> > > > > > >
> : > > > > >> > > > > > >
> : > > > > >> > > > > > > --
> : > > > > >> > > > > > > Sincerely yours
> : > > > > >> > > > > > > Mikhail Khludnev
> : > > > > >> > > > > > >
> : > > > > >> > > > > >
> : > > > > >> > > > >
> : > > > > >> > > > >
> : > > > > >> > > > >
> : > > > > >> > > > > --
> : > > > > >> > > > > Sincerely yours
> : > > > > >> > > > > Mikhail Khludnev
> : > > > > >> > > > >
> : > > > > >> > > >
> : > > > > >> > >
> : > > > > >> > >
> : > > > > >> > >
> : > > > > >> > > --
> : > > > > >> > > Sincerely yours
> : > > > > >> > > Mikhail Khludnev
> : > > > > >> > >
> : > > > > >> >
> : > > > > >>
> : > > > > >>
> : > > > > >>
> : > > > > >> --
> : > > > > >> Sincerely yours
> : > > > > >> Mikhail Khludnev
> : > > > > >>
> : > > > > >
> : > > > > >
> : > > > >
> : > > >
> : > > >
> : > > >
> : > > > --
> : > > > Sincerely yours
> : > > > Mikhail Khludnev
> : > > >
> : > >
> : >
> : >
> : >
> : > --
> : > Sincerely yours
> : > Mikhail Khludnev
> : >
> :
>
> -Hoss
> http://www.lucidworks.com/
>
Re: custom response writer which extends RawResponseWriter fails
when shards > 1
Posted by Chris Hostetter <ho...@fucit.org>.
Invariant really means "invariant" ... nothing can change them
In the case of "wt" this may seem weird and unhelpful, but the code that
handles defaults/appends/invariants is ignorant of what the params are.
Since your writting custom code anyway, my suggestion would be that
perhaps you could make your custom ResponseWriter delegate to the javabin
responsewriter if/when you see that this is an "isShard=true" request?
: Date: Thu, 19 Apr 2018 18:42:58 +0100
: From: Lee Carroll <le...@googlemail.com>
: Reply-To: solr-user@lucene.apache.org
: To: solr-user@lucene.apache.org
: Subject: Re: custom response writer which extends RawResponseWriter fails when
: shards > 1
:
: Hi,
:
: I rewrote all of my tests to use SolrCloudTestCase rather than SolrTestCaseJ4
: and was able to replicate the responsewriter issue and debug with a sharded
: collection. It turned out the issue was not with my response writer really
: but rather my config.
:
: <requestHandler name="/content" class="solr.SearchHandler">
: ....
:
: <lst name="invariants">
: <str name="wt">content</str>
: </lst>
:
: </requestHandler>
:
: In cloud mode having wt as an invariant breaks the collation of results
: from shards. Now I'm sure this is a common mistake which I've repeated
: (blush) but I do sort of want to actually implement my request handler in
: this way. Is their a way to have a request handler support a single
: response writer but still work in cloud mode ?
:
: Could this be considered a bug ?
:
: Lee C
:
: On 18 April 2018 at 13:13, Mikhail Khludnev <mk...@apache.org> wrote:
:
: > Injecting headers might require deeper customisation up to establishing own
: > filter or so.
: > Speaking regarding your own WT, there might be some issues because usually
: > it's not a big deal to use one wt for responding user query like (wt=csv)
: > and wt=javabin in internal communication between aggregator and slaves like
: > it happens in wt=csv query.
: >
: > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll <lee.a.carroll@googlemail.com
: > >
: > wrote:
: >
: > > Inventive. I need to control content-type of the response from the
: > document
: > > field value. I have the actual content field and the content-type field
: > to
: > > use configured in the response writer. I've just noticed that the xslt
: > > transformer allows you to do this but not controlled by document values.
: > I
: > > may also need to set some headers based on content-type and perhaps
: > content
: > > size, accept ranges comes to mind. Although I might be getting ahead of
: > > myself.
: > >
: > >
: > >
: > > On 18 April 2018 at 12:05, Mikhail Khludnev <mk...@apache.org> wrote:
: > >
: > > > well ..
: > > > what if
: > > > http://localhost:8983/solr/images/select?fl=content&q=id:
: > > 1&start=1&wt=csv&
: > > > csv.separator=&csv.encapsulator&csv.null=null
: > > > ?
: > > >
: > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll <
: > > lee.a.carroll@googlemail.com
: > > > >
: > > > wrote:
: > > >
: > > > > sorry cut n paste error i'd get
: > > > >
: > > > > {
: > > > > "responseHeader":{
: > > > > "zkConnected":true,
: > > > > "status":0,
: > > > > "QTime":0,
: > > > > "params":{
: > > > > "q":"*:*",
: > > > > "fl":"content",
: > > > > "rows":"1"}},
: > > > > "response":{"numFound":1,"start":0,"docs":[
: > > > > {
: > > > > "content":"my-content-value"}]
: > > > > }}
: > > > >
: > > > >
: > > > > but you get my point
: > > > >
: > > > >
: > > > >
: > > > > On 18 April 2018 at 11:13, Lee Carroll <lee.a.carroll@googlemail.com
: > >
: > > > > wrote:
: > > > >
: > > > > > for http://localhost:8983/solr/images/select?fl=content&q=id:
: > > 1&start=1
: > > > > >
: > > > > > I'd get
: > > > > >
: > > > > > {
: > > > > > "responseHeader":{
: > > > > > "zkConnected":true,
: > > > > > "status":0,
: > > > > > "QTime":1,
: > > > > > "params":{
: > > > > > "q":"*:*",
: > > > > > "_":"1524046333220"}},
: > > > > > "response":{"numFound":1,"start":0,"docs":[
: > > > > > {
: > > > > > "id":"1",
: > > > > > "content":"my-content-value",
: > > > > > "*content-type*":"text/plain"}]
: > > > > > }}
: > > > > >
: > > > > > when i want
: > > > > >
: > > > > > my-content-value
: > > > > >
: > > > > >
: > > > > >
: > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev <mk...@apache.org>
: > wrote:
: > > > > >
: > > > > >> Lee, from this description I don see why it can't be addressed by
: > > > > fl,rows
: > > > > >> params. What makes it different form the typical Solr usage?
: > > > > >>
: > > > > >>
: > > > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll <
: > > > > >> lee.a.carroll@googlemail.com>
: > > > > >> wrote:
: > > > > >>
: > > > > >> > Sure, we want to return a single field's value for the top
: > > matching
: > > > > >> > document for a given query. Bare content rather than a full
: > search
: > > > > >> result
: > > > > >> > listing.
: > > > > >> >
: > > > > >> > To be concrete:
: > > > > >> >
: > > > > >> > For a schema of fields id [unique key],
: > > > content[stored],content-type[
: > > > > >> > stored]
: > > > > >> > For a request:
: > > > > >> >
: > > > > >> > 1. Request URL:
: > > > > >> > https://localhost/solr/content?q=id:1
: > > > > >> > 2. Request Method:
: > > > > >> > GET
: > > > > >> >
: > > > > >> > We get a response
: > > > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type:
: > [content-type
: > > > > value]
: > > > > >> >
: > > > > >> > and the body to be the raw value of content
: > > > > >> >
: > > > > >> > In short clients consume directly the most relevant "content"
: > > > returned
: > > > > >> from
: > > > > >> > solr queries they construct.
: > > > > >> >
: > > > > >> > Naively I've implemented a subclass of RawResponseWriter which
: > > takes
: > > > > the
: > > > > >> > first docs values and adds them to the appended "content"
: > stream.
: > > > > >> Should I
: > > > > >> > selectively add the content stream depending on if this is the
: > > final
: > > > > >> > aggregation of cloud results (and provide a base class writer to
: > > act
: > > > > if
: > > > > >> > not), if so how do I know its the final aggregation. Or is
: > adding
: > > > the
: > > > > >> > content stream within the response writer a bad idea. Should
: > that
: > > be
: > > > > >> being
: > > > > >> > added to the response somewhere else?
: > > > > >> >
: > > > > >> > Failing all of the above is asking about response writer an X /
: > Y
: > > > > >> problem.
: > > > > >> > Is their a better way to achieve the above. I'd looked at
: > > > transforming
: > > > > >> > response xml but that seemed not to offer a complete bare slate.
: > > > > >> >
: > > > > >> > Cheers Lee C
: > > > > >> >
: > > > > >> >
: > > > > >> > On 17 April 2018 at 21:36, Mikhail Khludnev <mk...@apache.org>
: > > > wrote:
: > > > > >> >
: > > > > >> > > In distributed search response writer is used twice
: > > > > >> > > https://lucene.apache.org/solr/guide/7_1/distributed-
: > > > requests.html
: > > > > >> > > once slave node that's where response writer yields "json"
: > > content
: > > > > >> and it
: > > > > >> > > upset aggregator node which is expect only javabin.
: > > > > >> > > I hardly can comment on rrw, it's probably used for responding
: > > > > >> separate
: > > > > >> > > files in distrib=false mode.
: > > > > >> > > You can start from describing why you need to create own
: > > response
: > > > > >> writer.
: > > > > >> > >
: > > > > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <
: > > > > >> > lee.a.carroll@googlemail.com
: > > > > >> > > >
: > > > > >> > > wrote:
: > > > > >> > >
: > > > > >> > > > Ok. My expectation was the response writer would not be used
: > > > until
: > > > > >> the
: > > > > >> > > > final serialization of the result. If my response writer
: > > breaks
: > > > > the
: > > > > >> > > > response writer contract, exactly the way rawResponseWriter
: > > does
: > > > > and
: > > > > >> > just
: > > > > >> > > > out puts a filed value how does that work? Does
: > > > rawResponseWriter
: > > > > >> > support
: > > > > >> > > > cloud mode?
: > > > > >> > > >
: > > > > >> > > >
: > > > > >> > > >
: > > > > >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev <
: > mkhl@apache.org>
: > > > > >> wrote:
: > > > > >> > > >
: > > > > >> > > > > That's what should happen.
: > > > > >> > > > >
: > > > > >> > > > > Expected mime type application/octet-stream but got
: > > > > >> application/json.
: > > > > >> > > > >
: > > > > >> > > > > Distributed search coordinator expect to merge slave
: > > responses
: > > > > in
: > > > > >> > > javabin
: > > > > >> > > > > format. But slave's wt indicated json.
: > > > > >> > > > > As far as I know only javabin might be used to distributed
: > > > > search
: > > > > >> > > > > underneath. Coordinator itself might yield json.
: > > > > >> > > > >
: > > > > >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
: > > > > >> > > > lee.a.carroll@googlemail.com
: > > > > >> > > > > >
: > > > > >> > > > > wrote:
: > > > > >> > > > >
: > > > > >> > > > > > Sure
: > > > > >> > > > > >
: > > > > >> > > > > > with 1 shard 1 replica this request works fine
: > > > > >> > > > > >
: > > > > >> > > > > > 1. Request URL:
: > > > > >> > > > > > http://localhost:8983/solr/images/image?q=id:1
: > > > > >> > > > > > 2. Request Method:
: > > > > >> > > > > > GET
: > > > > >> > > > > > 3. Status Code:
: > > > > >> > > > > > 200 OK
: > > > > >> > > > > >
: > > > > >> > > > > > logs are clean
: > > > > >> > > > > >
: > > > > >> > > > > > with 2 shards 2 replicas the same request fails and in
: > the
: > > > > logs
: > > > > >> > > > > >
: > > > > >> > > > > >
: > > > > >> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2
: > > > > r:core_node7
: > > > > >> > > > > > x:images_shard2_replica_n4]
: > org.apache.solr.core.SolrCore;
: > > > > >> > > > > > [images_shard2_replica_n4] webapp=/solr path=/image
: > > > > >> > > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
: > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
: > > > > >> > > > > > http://10.224.30.207:8983/
: > solr/images_shard2_replica_n4/
: > > > > >> > > > > > |http://10.224.30.207:7574/
: > solr/images_shard2_replica_n6/
: > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
: > > > 1523971232039&isShard=true&wt=
: > > > > >> > javabin}
: > > > > >> > > > > > hits=0 status=0 QTime=0
: > > > > >> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1
: > > > > r:core_node3
: > > > > >> > > > > > x:images_shard1_replica_n1]
: > org.apache.solr.common.SolrExc
: > > > > >> eption;
: > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
: > > > > >> > > RemoteSolrException:
: > > > > >> > > > > > Error
: > > > > >> > > > > > from server at http://10.224.30.207:8983/
: > > > > >> > > solr/images_shard2_replica_n4
: > > > > >> > > > :
: > > > > >> > > > > > Expected mime type application/octet-stream but got
: > > > > >> > application/json.
: > > > > >> > > > > > at
: > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
: > > > > >> > > > > > executeMethod(HttpSolrClient.java:607)
: > > > > >> > > > > > at
: > > > > >> > > > > > org.apache.solr.client.solrj.
: > impl.HttpSolrClient.request(
: > > > > >> > > > > > HttpSolrClient.java:255)
: > > > > >> > > > > > at
: > > > > >> > > > > > org.apache.solr.client.solrj.
: > impl.HttpSolrClient.request(
: > > > > >> > > > > > HttpSolrClient.java:244)
: > > > > >> > > > > > at
: > > > > >> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.
: > > > > >> > > > > > doRequest(LBHttpSolrClient.java:483)
: > > > > >> > > > > > at
: > > > > >> > > > > > org.apache.solr.client.solrj.
: > > impl.LBHttpSolrClient.request(
: > > > > >> > > > > > LBHttpSolrClient.java:413)
: > > > > >> > > > > > at
: > > > > >> > > > > > org.apache.solr.handler.component.
: > > HttpShardHandlerFactory.
: > > > > >> > > > > > makeLoadBalancedRequest(HttpShardHandlerFactory.java:
: > 273)
: > > > > >> > > > > > at
: > > > > >> > > > > > org.apache.solr.handler.component.HttpShardHandler.
: > > > > >> > lambda$submit$0(
: > > > > >> > > > > > HttpShardHandler.java:175)
: > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
: > > java:266)
: > > > > >> > > > > > at java.util.concurrent.Executors$RunnableAdapter.
: > > > > >> > > > > call(Executors.java:511)
: > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
: > > java:266)
: > > > > >> > > > > > at
: > > > > >> > > > > > com.codahale.metrics.InstrumentedExecutorService$
: > > > > >> > > > > InstrumentedRunnable.run(
: > > > > >> > > > > > InstrumentedExecutorService.java:176)
: > > > > >> > > > > > at
: > > > > >> > > > > > org.apache.solr.common.util.ExecutorUtil$
: > > > > >> > MDCAwareThreadPoolExecutor.
: > > > > >> > > > > > lambda$execute$0(ExecutorUtil.java:188)
: > > > > >> > > > > > at
: > > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
: > > > > >> > > > > > ThreadPoolExecutor.java:1142)
: > > > > >> > > > > > at
: > > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
: > > > > >> > > > > > ThreadPoolExecutor.java:617)
: > > > > >> > > > > > at java.lang.Thread.run(Thread.java:745)
: > > > > >> > > > > >
: > > > > >> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1
: > > > > r:core_node3
: > > > > >> > > > > > x:images_shard1_replica_n1]
: > org.apache.solr.core.SolrCore;
: > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
: > > > > >> > params={q=id:1}
: > > > > >> > > > > > status=200 QTime=17
: > > > > >> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1
: > > > > r:core_node3
: > > > > >> > > > > > x:images_shard1_replica_n1]
: > org.apache.solr.core.SolrCore;
: > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
: > > > > >> > > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
: > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
: > > > > >> > > > > > http://10.224.30.207:8983/
: > solr/images_shard1_replica_n1/
: > > > > >> > > > > > |http://10.224.30.207:7574/
: > solr/images_shard1_replica_n2/
: > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
: > > > 1523971232039&isShard=true&wt=
: > > > > >> > javabin}
: > > > > >> > > > > > hits=1 status=0 QTime=2
: > > > > >> > > > > >
: > > > > >> > > > > >
: > > > > >> > > > > > I've implemented getcontenttype simply as
: > > > > >> > > > > >
: > > > > >> > > > > > @Override
: > > > > >> > > > > > public String getContentType(SolrQueryRequest request,
: > > > > >> > > > > > SolrQueryResponse response) {
: > > > > >> > > > > >
: > > > > >> > > > > > return "application/json;charset=utf-8";
: > > > > >> > > > > > }
: > > > > >> > > > > >
: > > > > >> > > > > >
: > > > > >> > > > > >
: > > > > >> > > > > >
: > > > > >> > > > > >
: > > > > >> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev <
: > > > mkhl@apache.org>
: > > > > >> > wrote:
: > > > > >> > > > > >
: > > > > >> > > > > > > Lee,
: > > > > >> > > > > > > It's worth to send a stacktrace for such kind of
: > > > inquiries.
: > > > > >> > > > > > > I guess it goes from QueryComponent.mergeIds() or so.
: > > > Shard
: > > > > >> > > response
: > > > > >> > > > > > should
: > > > > >> > > > > > > contains <uniqueKey> from schema.xml field.
: > > > > >> > > > > > > I encounter something like this while troubleshooting
: > > > > >> > > > > > > https://lucene.apache.org/
: > solr/guide/6_6/transforming-
: > > > > >> > > > > > > result-documents.html#TransformingResultDocuments-
: > > > > >> > > > > > > CoresandCollectionsinSolrCloud
: > > > > >> > > > > > >
: > > > > >> > > > > > >
: > > > > >> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
: > > > > >> > > > > > lee.a.carroll@googlemail.com
: > > > > >> > > > > > > >
: > > > > >> > > > > > > wrote:
: > > > > >> > > > > > >
: > > > > >> > > > > > > > I've created a custom response writer which extends
: > > > > >> > > > > RawResponseWriter.
: > > > > >> > > > > > > The
: > > > > >> > > > > > > > basic operation is to output a single field value
: > from
: > > > the
: > > > > >> top
: > > > > >> > > > > matching
: > > > > >> > > > > > > doc
: > > > > >> > > > > > > > as the entire response. This works when shards = 1
: > but
: > > > > fails
: > > > > >> > when
: > > > > >> > > > > > shards
: > > > > >> > > > > > > > are greater than 1.
: > > > > >> > > > > > > >
: > > > > >> > > > > > > > I throw an error if the field in question is missing
: > > > from
: > > > > >> the
: > > > > >> > top
: > > > > >> > > > > doc.
: > > > > >> > > > > > > This
: > > > > >> > > > > > > > happens when individual shards are being searched
: > and
: > > > only
: > > > > >> id
: > > > > >> > and
: > > > > >> > > > > score
: > > > > >> > > > > > > are
: > > > > >> > > > > > > > returned. I'm sure I've committed a basic error.
: > > > > >> > > > > > > >
: > > > > >> > > > > > > > Lee C
: > > > > >> > > > > > > >
: > > > > >> > > > > > >
: > > > > >> > > > > > >
: > > > > >> > > > > > >
: > > > > >> > > > > > > --
: > > > > >> > > > > > > Sincerely yours
: > > > > >> > > > > > > Mikhail Khludnev
: > > > > >> > > > > > >
: > > > > >> > > > > >
: > > > > >> > > > >
: > > > > >> > > > >
: > > > > >> > > > >
: > > > > >> > > > > --
: > > > > >> > > > > Sincerely yours
: > > > > >> > > > > Mikhail Khludnev
: > > > > >> > > > >
: > > > > >> > > >
: > > > > >> > >
: > > > > >> > >
: > > > > >> > >
: > > > > >> > > --
: > > > > >> > > Sincerely yours
: > > > > >> > > Mikhail Khludnev
: > > > > >> > >
: > > > > >> >
: > > > > >>
: > > > > >>
: > > > > >>
: > > > > >> --
: > > > > >> Sincerely yours
: > > > > >> Mikhail Khludnev
: > > > > >>
: > > > > >
: > > > > >
: > > > >
: > > >
: > > >
: > > >
: > > > --
: > > > Sincerely yours
: > > > Mikhail Khludnev
: > > >
: > >
: >
: >
: >
: > --
: > Sincerely yours
: > Mikhail Khludnev
: >
:
-Hoss
http://www.lucidworks.com/
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Lee Carroll <le...@googlemail.com>.
Hi,
I rewrote all of my tests to use SolrCloudTestCase rather than SolrTestCaseJ4
and was able to replicate the responsewriter issue and debug with a sharded
collection. It turned out the issue was not with my response writer really
but rather my config.
<requestHandler name="/content" class="solr.SearchHandler">
....
<lst name="invariants">
<str name="wt">content</str>
</lst>
</requestHandler>
In cloud mode having wt as an invariant breaks the collation of results
from shards. Now I'm sure this is a common mistake which I've repeated
(blush) but I do sort of want to actually implement my request handler in
this way. Is their a way to have a request handler support a single
response writer but still work in cloud mode ?
Could this be considered a bug ?
Lee C
On 18 April 2018 at 13:13, Mikhail Khludnev <mk...@apache.org> wrote:
> Injecting headers might require deeper customisation up to establishing own
> filter or so.
> Speaking regarding your own WT, there might be some issues because usually
> it's not a big deal to use one wt for responding user query like (wt=csv)
> and wt=javabin in internal communication between aggregator and slaves like
> it happens in wt=csv query.
>
> On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll <lee.a.carroll@googlemail.com
> >
> wrote:
>
> > Inventive. I need to control content-type of the response from the
> document
> > field value. I have the actual content field and the content-type field
> to
> > use configured in the response writer. I've just noticed that the xslt
> > transformer allows you to do this but not controlled by document values.
> I
> > may also need to set some headers based on content-type and perhaps
> content
> > size, accept ranges comes to mind. Although I might be getting ahead of
> > myself.
> >
> >
> >
> > On 18 April 2018 at 12:05, Mikhail Khludnev <mk...@apache.org> wrote:
> >
> > > well ..
> > > what if
> > > http://localhost:8983/solr/images/select?fl=content&q=id:
> > 1&start=1&wt=csv&
> > > csv.separator=&csv.encapsulator&csv.null=null
> > > ?
> > >
> > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll <
> > lee.a.carroll@googlemail.com
> > > >
> > > wrote:
> > >
> > > > sorry cut n paste error i'd get
> > > >
> > > > {
> > > > "responseHeader":{
> > > > "zkConnected":true,
> > > > "status":0,
> > > > "QTime":0,
> > > > "params":{
> > > > "q":"*:*",
> > > > "fl":"content",
> > > > "rows":"1"}},
> > > > "response":{"numFound":1,"start":0,"docs":[
> > > > {
> > > > "content":"my-content-value"}]
> > > > }}
> > > >
> > > >
> > > > but you get my point
> > > >
> > > >
> > > >
> > > > On 18 April 2018 at 11:13, Lee Carroll <lee.a.carroll@googlemail.com
> >
> > > > wrote:
> > > >
> > > > > for http://localhost:8983/solr/images/select?fl=content&q=id:
> > 1&start=1
> > > > >
> > > > > I'd get
> > > > >
> > > > > {
> > > > > "responseHeader":{
> > > > > "zkConnected":true,
> > > > > "status":0,
> > > > > "QTime":1,
> > > > > "params":{
> > > > > "q":"*:*",
> > > > > "_":"1524046333220"}},
> > > > > "response":{"numFound":1,"start":0,"docs":[
> > > > > {
> > > > > "id":"1",
> > > > > "content":"my-content-value",
> > > > > "*content-type*":"text/plain"}]
> > > > > }}
> > > > >
> > > > > when i want
> > > > >
> > > > > my-content-value
> > > > >
> > > > >
> > > > >
> > > > > On 18 April 2018 at 10:55, Mikhail Khludnev <mk...@apache.org>
> wrote:
> > > > >
> > > > >> Lee, from this description I don see why it can't be addressed by
> > > > fl,rows
> > > > >> params. What makes it different form the typical Solr usage?
> > > > >>
> > > > >>
> > > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll <
> > > > >> lee.a.carroll@googlemail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Sure, we want to return a single field's value for the top
> > matching
> > > > >> > document for a given query. Bare content rather than a full
> search
> > > > >> result
> > > > >> > listing.
> > > > >> >
> > > > >> > To be concrete:
> > > > >> >
> > > > >> > For a schema of fields id [unique key],
> > > content[stored],content-type[
> > > > >> > stored]
> > > > >> > For a request:
> > > > >> >
> > > > >> > 1. Request URL:
> > > > >> > https://localhost/solr/content?q=id:1
> > > > >> > 2. Request Method:
> > > > >> > GET
> > > > >> >
> > > > >> > We get a response
> > > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type:
> [content-type
> > > > value]
> > > > >> >
> > > > >> > and the body to be the raw value of content
> > > > >> >
> > > > >> > In short clients consume directly the most relevant "content"
> > > returned
> > > > >> from
> > > > >> > solr queries they construct.
> > > > >> >
> > > > >> > Naively I've implemented a subclass of RawResponseWriter which
> > takes
> > > > the
> > > > >> > first docs values and adds them to the appended "content"
> stream.
> > > > >> Should I
> > > > >> > selectively add the content stream depending on if this is the
> > final
> > > > >> > aggregation of cloud results (and provide a base class writer to
> > act
> > > > if
> > > > >> > not), if so how do I know its the final aggregation. Or is
> adding
> > > the
> > > > >> > content stream within the response writer a bad idea. Should
> that
> > be
> > > > >> being
> > > > >> > added to the response somewhere else?
> > > > >> >
> > > > >> > Failing all of the above is asking about response writer an X /
> Y
> > > > >> problem.
> > > > >> > Is their a better way to achieve the above. I'd looked at
> > > transforming
> > > > >> > response xml but that seemed not to offer a complete bare slate.
> > > > >> >
> > > > >> > Cheers Lee C
> > > > >> >
> > > > >> >
> > > > >> > On 17 April 2018 at 21:36, Mikhail Khludnev <mk...@apache.org>
> > > wrote:
> > > > >> >
> > > > >> > > In distributed search response writer is used twice
> > > > >> > > https://lucene.apache.org/solr/guide/7_1/distributed-
> > > requests.html
> > > > >> > > once slave node that's where response writer yields "json"
> > content
> > > > >> and it
> > > > >> > > upset aggregator node which is expect only javabin.
> > > > >> > > I hardly can comment on rrw, it's probably used for responding
> > > > >> separate
> > > > >> > > files in distrib=false mode.
> > > > >> > > You can start from describing why you need to create own
> > response
> > > > >> writer.
> > > > >> > >
> > > > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <
> > > > >> > lee.a.carroll@googlemail.com
> > > > >> > > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Ok. My expectation was the response writer would not be used
> > > until
> > > > >> the
> > > > >> > > > final serialization of the result. If my response writer
> > breaks
> > > > the
> > > > >> > > > response writer contract, exactly the way rawResponseWriter
> > does
> > > > and
> > > > >> > just
> > > > >> > > > out puts a filed value how does that work? Does
> > > rawResponseWriter
> > > > >> > support
> > > > >> > > > cloud mode?
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev <
> mkhl@apache.org>
> > > > >> wrote:
> > > > >> > > >
> > > > >> > > > > That's what should happen.
> > > > >> > > > >
> > > > >> > > > > Expected mime type application/octet-stream but got
> > > > >> application/json.
> > > > >> > > > >
> > > > >> > > > > Distributed search coordinator expect to merge slave
> > responses
> > > > in
> > > > >> > > javabin
> > > > >> > > > > format. But slave's wt indicated json.
> > > > >> > > > > As far as I know only javabin might be used to distributed
> > > > search
> > > > >> > > > > underneath. Coordinator itself might yield json.
> > > > >> > > > >
> > > > >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
> > > > >> > > > lee.a.carroll@googlemail.com
> > > > >> > > > > >
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Sure
> > > > >> > > > > >
> > > > >> > > > > > with 1 shard 1 replica this request works fine
> > > > >> > > > > >
> > > > >> > > > > > 1. Request URL:
> > > > >> > > > > > http://localhost:8983/solr/images/image?q=id:1
> > > > >> > > > > > 2. Request Method:
> > > > >> > > > > > GET
> > > > >> > > > > > 3. Status Code:
> > > > >> > > > > > 200 OK
> > > > >> > > > > >
> > > > >> > > > > > logs are clean
> > > > >> > > > > >
> > > > >> > > > > > with 2 shards 2 replicas the same request fails and in
> the
> > > > logs
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2
> > > > r:core_node7
> > > > >> > > > > > x:images_shard2_replica_n4]
> org.apache.solr.core.SolrCore;
> > > > >> > > > > > [images_shard2_replica_n4] webapp=/solr path=/image
> > > > >> > > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > > >> > > > > > http://10.224.30.207:8983/
> solr/images_shard2_replica_n4/
> > > > >> > > > > > |http://10.224.30.207:7574/
> solr/images_shard2_replica_n6/
> > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
> > > 1523971232039&isShard=true&wt=
> > > > >> > javabin}
> > > > >> > > > > > hits=0 status=0 QTime=0
> > > > >> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1
> > > > r:core_node3
> > > > >> > > > > > x:images_shard1_replica_n1]
> org.apache.solr.common.SolrExc
> > > > >> eption;
> > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
> > > > >> > > RemoteSolrException:
> > > > >> > > > > > Error
> > > > >> > > > > > from server at http://10.224.30.207:8983/
> > > > >> > > solr/images_shard2_replica_n4
> > > > >> > > > :
> > > > >> > > > > > Expected mime type application/octet-stream but got
> > > > >> > application/json.
> > > > >> > > > > > at
> > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
> > > > >> > > > > > executeMethod(HttpSolrClient.java:607)
> > > > >> > > > > > at
> > > > >> > > > > > org.apache.solr.client.solrj.
> impl.HttpSolrClient.request(
> > > > >> > > > > > HttpSolrClient.java:255)
> > > > >> > > > > > at
> > > > >> > > > > > org.apache.solr.client.solrj.
> impl.HttpSolrClient.request(
> > > > >> > > > > > HttpSolrClient.java:244)
> > > > >> > > > > > at
> > > > >> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > > > >> > > > > > doRequest(LBHttpSolrClient.java:483)
> > > > >> > > > > > at
> > > > >> > > > > > org.apache.solr.client.solrj.
> > impl.LBHttpSolrClient.request(
> > > > >> > > > > > LBHttpSolrClient.java:413)
> > > > >> > > > > > at
> > > > >> > > > > > org.apache.solr.handler.component.
> > HttpShardHandlerFactory.
> > > > >> > > > > > makeLoadBalancedRequest(HttpShardHandlerFactory.java:
> 273)
> > > > >> > > > > > at
> > > > >> > > > > > org.apache.solr.handler.component.HttpShardHandler.
> > > > >> > lambda$submit$0(
> > > > >> > > > > > HttpShardHandler.java:175)
> > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
> > java:266)
> > > > >> > > > > > at java.util.concurrent.Executors$RunnableAdapter.
> > > > >> > > > > call(Executors.java:511)
> > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
> > java:266)
> > > > >> > > > > > at
> > > > >> > > > > > com.codahale.metrics.InstrumentedExecutorService$
> > > > >> > > > > InstrumentedRunnable.run(
> > > > >> > > > > > InstrumentedExecutorService.java:176)
> > > > >> > > > > > at
> > > > >> > > > > > org.apache.solr.common.util.ExecutorUtil$
> > > > >> > MDCAwareThreadPoolExecutor.
> > > > >> > > > > > lambda$execute$0(ExecutorUtil.java:188)
> > > > >> > > > > > at
> > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > >> > > > > > ThreadPoolExecutor.java:1142)
> > > > >> > > > > > at
> > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > >> > > > > > ThreadPoolExecutor.java:617)
> > > > >> > > > > > at java.lang.Thread.run(Thread.java:745)
> > > > >> > > > > >
> > > > >> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1
> > > > r:core_node3
> > > > >> > > > > > x:images_shard1_replica_n1]
> org.apache.solr.core.SolrCore;
> > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> > > > >> > params={q=id:1}
> > > > >> > > > > > status=200 QTime=17
> > > > >> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1
> > > > r:core_node3
> > > > >> > > > > > x:images_shard1_replica_n1]
> org.apache.solr.core.SolrCore;
> > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> > > > >> > > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > > >> > > > > > http://10.224.30.207:8983/
> solr/images_shard1_replica_n1/
> > > > >> > > > > > |http://10.224.30.207:7574/
> solr/images_shard1_replica_n2/
> > > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
> > > 1523971232039&isShard=true&wt=
> > > > >> > javabin}
> > > > >> > > > > > hits=1 status=0 QTime=2
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > I've implemented getcontenttype simply as
> > > > >> > > > > >
> > > > >> > > > > > @Override
> > > > >> > > > > > public String getContentType(SolrQueryRequest request,
> > > > >> > > > > > SolrQueryResponse response) {
> > > > >> > > > > >
> > > > >> > > > > > return "application/json;charset=utf-8";
> > > > >> > > > > > }
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev <
> > > mkhl@apache.org>
> > > > >> > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Lee,
> > > > >> > > > > > > It's worth to send a stacktrace for such kind of
> > > inquiries.
> > > > >> > > > > > > I guess it goes from QueryComponent.mergeIds() or so.
> > > Shard
> > > > >> > > response
> > > > >> > > > > > should
> > > > >> > > > > > > contains <uniqueKey> from schema.xml field.
> > > > >> > > > > > > I encounter something like this while troubleshooting
> > > > >> > > > > > > https://lucene.apache.org/
> solr/guide/6_6/transforming-
> > > > >> > > > > > > result-documents.html#TransformingResultDocuments-
> > > > >> > > > > > > CoresandCollectionsinSolrCloud
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
> > > > >> > > > > > lee.a.carroll@googlemail.com
> > > > >> > > > > > > >
> > > > >> > > > > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > I've created a custom response writer which extends
> > > > >> > > > > RawResponseWriter.
> > > > >> > > > > > > The
> > > > >> > > > > > > > basic operation is to output a single field value
> from
> > > the
> > > > >> top
> > > > >> > > > > matching
> > > > >> > > > > > > doc
> > > > >> > > > > > > > as the entire response. This works when shards = 1
> but
> > > > fails
> > > > >> > when
> > > > >> > > > > > shards
> > > > >> > > > > > > > are greater than 1.
> > > > >> > > > > > > >
> > > > >> > > > > > > > I throw an error if the field in question is missing
> > > from
> > > > >> the
> > > > >> > top
> > > > >> > > > > doc.
> > > > >> > > > > > > This
> > > > >> > > > > > > > happens when individual shards are being searched
> and
> > > only
> > > > >> id
> > > > >> > and
> > > > >> > > > > score
> > > > >> > > > > > > are
> > > > >> > > > > > > > returned. I'm sure I've committed a basic error.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Lee C
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > --
> > > > >> > > > > > > Sincerely yours
> > > > >> > > > > > > Mikhail Khludnev
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > --
> > > > >> > > > > Sincerely yours
> > > > >> > > > > Mikhail Khludnev
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > Sincerely yours
> > > > >> > > Mikhail Khludnev
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Sincerely yours
> > > > >> Mikhail Khludnev
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > >
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Mikhail Khludnev <mk...@apache.org>.
Injecting headers might require deeper customisation up to establishing own
filter or so.
Speaking regarding your own WT, there might be some issues because usually
it's not a big deal to use one wt for responding user query like (wt=csv)
and wt=javabin in internal communication between aggregator and slaves like
it happens in wt=csv query.
On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll <le...@googlemail.com>
wrote:
> Inventive. I need to control content-type of the response from the document
> field value. I have the actual content field and the content-type field to
> use configured in the response writer. I've just noticed that the xslt
> transformer allows you to do this but not controlled by document values. I
> may also need to set some headers based on content-type and perhaps content
> size, accept ranges comes to mind. Although I might be getting ahead of
> myself.
>
>
>
> On 18 April 2018 at 12:05, Mikhail Khludnev <mk...@apache.org> wrote:
>
> > well ..
> > what if
> > http://localhost:8983/solr/images/select?fl=content&q=id:
> 1&start=1&wt=csv&
> > csv.separator=&csv.encapsulator&csv.null=null
> > ?
> >
> > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll <
> lee.a.carroll@googlemail.com
> > >
> > wrote:
> >
> > > sorry cut n paste error i'd get
> > >
> > > {
> > > "responseHeader":{
> > > "zkConnected":true,
> > > "status":0,
> > > "QTime":0,
> > > "params":{
> > > "q":"*:*",
> > > "fl":"content",
> > > "rows":"1"}},
> > > "response":{"numFound":1,"start":0,"docs":[
> > > {
> > > "content":"my-content-value"}]
> > > }}
> > >
> > >
> > > but you get my point
> > >
> > >
> > >
> > > On 18 April 2018 at 11:13, Lee Carroll <le...@googlemail.com>
> > > wrote:
> > >
> > > > for http://localhost:8983/solr/images/select?fl=content&q=id:
> 1&start=1
> > > >
> > > > I'd get
> > > >
> > > > {
> > > > "responseHeader":{
> > > > "zkConnected":true,
> > > > "status":0,
> > > > "QTime":1,
> > > > "params":{
> > > > "q":"*:*",
> > > > "_":"1524046333220"}},
> > > > "response":{"numFound":1,"start":0,"docs":[
> > > > {
> > > > "id":"1",
> > > > "content":"my-content-value",
> > > > "*content-type*":"text/plain"}]
> > > > }}
> > > >
> > > > when i want
> > > >
> > > > my-content-value
> > > >
> > > >
> > > >
> > > > On 18 April 2018 at 10:55, Mikhail Khludnev <mk...@apache.org> wrote:
> > > >
> > > >> Lee, from this description I don see why it can't be addressed by
> > > fl,rows
> > > >> params. What makes it different form the typical Solr usage?
> > > >>
> > > >>
> > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll <
> > > >> lee.a.carroll@googlemail.com>
> > > >> wrote:
> > > >>
> > > >> > Sure, we want to return a single field's value for the top
> matching
> > > >> > document for a given query. Bare content rather than a full search
> > > >> result
> > > >> > listing.
> > > >> >
> > > >> > To be concrete:
> > > >> >
> > > >> > For a schema of fields id [unique key],
> > content[stored],content-type[
> > > >> > stored]
> > > >> > For a request:
> > > >> >
> > > >> > 1. Request URL:
> > > >> > https://localhost/solr/content?q=id:1
> > > >> > 2. Request Method:
> > > >> > GET
> > > >> >
> > > >> > We get a response
> > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: [content-type
> > > value]
> > > >> >
> > > >> > and the body to be the raw value of content
> > > >> >
> > > >> > In short clients consume directly the most relevant "content"
> > returned
> > > >> from
> > > >> > solr queries they construct.
> > > >> >
> > > >> > Naively I've implemented a subclass of RawResponseWriter which
> takes
> > > the
> > > >> > first docs values and adds them to the appended "content" stream.
> > > >> Should I
> > > >> > selectively add the content stream depending on if this is the
> final
> > > >> > aggregation of cloud results (and provide a base class writer to
> act
> > > if
> > > >> > not), if so how do I know its the final aggregation. Or is adding
> > the
> > > >> > content stream within the response writer a bad idea. Should that
> be
> > > >> being
> > > >> > added to the response somewhere else?
> > > >> >
> > > >> > Failing all of the above is asking about response writer an X / Y
> > > >> problem.
> > > >> > Is their a better way to achieve the above. I'd looked at
> > transforming
> > > >> > response xml but that seemed not to offer a complete bare slate.
> > > >> >
> > > >> > Cheers Lee C
> > > >> >
> > > >> >
> > > >> > On 17 April 2018 at 21:36, Mikhail Khludnev <mk...@apache.org>
> > wrote:
> > > >> >
> > > >> > > In distributed search response writer is used twice
> > > >> > > https://lucene.apache.org/solr/guide/7_1/distributed-
> > requests.html
> > > >> > > once slave node that's where response writer yields "json"
> content
> > > >> and it
> > > >> > > upset aggregator node which is expect only javabin.
> > > >> > > I hardly can comment on rrw, it's probably used for responding
> > > >> separate
> > > >> > > files in distrib=false mode.
> > > >> > > You can start from describing why you need to create own
> response
> > > >> writer.
> > > >> > >
> > > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <
> > > >> > lee.a.carroll@googlemail.com
> > > >> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Ok. My expectation was the response writer would not be used
> > until
> > > >> the
> > > >> > > > final serialization of the result. If my response writer
> breaks
> > > the
> > > >> > > > response writer contract, exactly the way rawResponseWriter
> does
> > > and
> > > >> > just
> > > >> > > > out puts a filed value how does that work? Does
> > rawResponseWriter
> > > >> > support
> > > >> > > > cloud mode?
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev <mk...@apache.org>
> > > >> wrote:
> > > >> > > >
> > > >> > > > > That's what should happen.
> > > >> > > > >
> > > >> > > > > Expected mime type application/octet-stream but got
> > > >> application/json.
> > > >> > > > >
> > > >> > > > > Distributed search coordinator expect to merge slave
> responses
> > > in
> > > >> > > javabin
> > > >> > > > > format. But slave's wt indicated json.
> > > >> > > > > As far as I know only javabin might be used to distributed
> > > search
> > > >> > > > > underneath. Coordinator itself might yield json.
> > > >> > > > >
> > > >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
> > > >> > > > lee.a.carroll@googlemail.com
> > > >> > > > > >
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Sure
> > > >> > > > > >
> > > >> > > > > > with 1 shard 1 replica this request works fine
> > > >> > > > > >
> > > >> > > > > > 1. Request URL:
> > > >> > > > > > http://localhost:8983/solr/images/image?q=id:1
> > > >> > > > > > 2. Request Method:
> > > >> > > > > > GET
> > > >> > > > > > 3. Status Code:
> > > >> > > > > > 200 OK
> > > >> > > > > >
> > > >> > > > > > logs are clean
> > > >> > > > > >
> > > >> > > > > > with 2 shards 2 replicas the same request fails and in the
> > > logs
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2
> > > r:core_node7
> > > >> > > > > > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore;
> > > >> > > > > > [images_shard2_replica_n4] webapp=/solr path=/image
> > > >> > > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > >> > > > > > http://10.224.30.207:8983/solr/images_shard2_replica_n4/
> > > >> > > > > > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/
> > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
> > 1523971232039&isShard=true&wt=
> > > >> > javabin}
> > > >> > > > > > hits=0 status=0 QTime=0
> > > >> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1
> > > r:core_node3
> > > >> > > > > > x:images_shard1_replica_n1] org.apache.solr.common.SolrExc
> > > >> eption;
> > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
> > > >> > > RemoteSolrException:
> > > >> > > > > > Error
> > > >> > > > > > from server at http://10.224.30.207:8983/
> > > >> > > solr/images_shard2_replica_n4
> > > >> > > > :
> > > >> > > > > > Expected mime type application/octet-stream but got
> > > >> > application/json.
> > > >> > > > > > at
> > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
> > > >> > > > > > executeMethod(HttpSolrClient.java:607)
> > > >> > > > > > at
> > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > > >> > > > > > HttpSolrClient.java:255)
> > > >> > > > > > at
> > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > > >> > > > > > HttpSolrClient.java:244)
> > > >> > > > > > at
> > > >> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > > >> > > > > > doRequest(LBHttpSolrClient.java:483)
> > > >> > > > > > at
> > > >> > > > > > org.apache.solr.client.solrj.
> impl.LBHttpSolrClient.request(
> > > >> > > > > > LBHttpSolrClient.java:413)
> > > >> > > > > > at
> > > >> > > > > > org.apache.solr.handler.component.
> HttpShardHandlerFactory.
> > > >> > > > > > makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
> > > >> > > > > > at
> > > >> > > > > > org.apache.solr.handler.component.HttpShardHandler.
> > > >> > lambda$submit$0(
> > > >> > > > > > HttpShardHandler.java:175)
> > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
> java:266)
> > > >> > > > > > at java.util.concurrent.Executors$RunnableAdapter.
> > > >> > > > > call(Executors.java:511)
> > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
> java:266)
> > > >> > > > > > at
> > > >> > > > > > com.codahale.metrics.InstrumentedExecutorService$
> > > >> > > > > InstrumentedRunnable.run(
> > > >> > > > > > InstrumentedExecutorService.java:176)
> > > >> > > > > > at
> > > >> > > > > > org.apache.solr.common.util.ExecutorUtil$
> > > >> > MDCAwareThreadPoolExecutor.
> > > >> > > > > > lambda$execute$0(ExecutorUtil.java:188)
> > > >> > > > > > at
> > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > >> > > > > > ThreadPoolExecutor.java:1142)
> > > >> > > > > > at
> > > >> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > >> > > > > > ThreadPoolExecutor.java:617)
> > > >> > > > > > at java.lang.Thread.run(Thread.java:745)
> > > >> > > > > >
> > > >> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1
> > > r:core_node3
> > > >> > > > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> > > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> > > >> > params={q=id:1}
> > > >> > > > > > status=200 QTime=17
> > > >> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1
> > > r:core_node3
> > > >> > > > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> > > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> > > >> > > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > >> > > > > > http://10.224.30.207:8983/solr/images_shard1_replica_n1/
> > > >> > > > > > |http://10.224.30.207:7574/solr/images_shard1_replica_n2/
> > > >> > > > > > &rows=10&version=2&q=id:1&NOW=
> > 1523971232039&isShard=true&wt=
> > > >> > javabin}
> > > >> > > > > > hits=1 status=0 QTime=2
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > I've implemented getcontenttype simply as
> > > >> > > > > >
> > > >> > > > > > @Override
> > > >> > > > > > public String getContentType(SolrQueryRequest request,
> > > >> > > > > > SolrQueryResponse response) {
> > > >> > > > > >
> > > >> > > > > > return "application/json;charset=utf-8";
> > > >> > > > > > }
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev <
> > mkhl@apache.org>
> > > >> > wrote:
> > > >> > > > > >
> > > >> > > > > > > Lee,
> > > >> > > > > > > It's worth to send a stacktrace for such kind of
> > inquiries.
> > > >> > > > > > > I guess it goes from QueryComponent.mergeIds() or so.
> > Shard
> > > >> > > response
> > > >> > > > > > should
> > > >> > > > > > > contains <uniqueKey> from schema.xml field.
> > > >> > > > > > > I encounter something like this while troubleshooting
> > > >> > > > > > > https://lucene.apache.org/solr/guide/6_6/transforming-
> > > >> > > > > > > result-documents.html#TransformingResultDocuments-
> > > >> > > > > > > CoresandCollectionsinSolrCloud
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
> > > >> > > > > > lee.a.carroll@googlemail.com
> > > >> > > > > > > >
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > I've created a custom response writer which extends
> > > >> > > > > RawResponseWriter.
> > > >> > > > > > > The
> > > >> > > > > > > > basic operation is to output a single field value from
> > the
> > > >> top
> > > >> > > > > matching
> > > >> > > > > > > doc
> > > >> > > > > > > > as the entire response. This works when shards = 1 but
> > > fails
> > > >> > when
> > > >> > > > > > shards
> > > >> > > > > > > > are greater than 1.
> > > >> > > > > > > >
> > > >> > > > > > > > I throw an error if the field in question is missing
> > from
> > > >> the
> > > >> > top
> > > >> > > > > doc.
> > > >> > > > > > > This
> > > >> > > > > > > > happens when individual shards are being searched and
> > only
> > > >> id
> > > >> > and
> > > >> > > > > score
> > > >> > > > > > > are
> > > >> > > > > > > > returned. I'm sure I've committed a basic error.
> > > >> > > > > > > >
> > > >> > > > > > > > Lee C
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > --
> > > >> > > > > > > Sincerely yours
> > > >> > > > > > > Mikhail Khludnev
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > --
> > > >> > > > > Sincerely yours
> > > >> > > > > Mikhail Khludnev
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > Sincerely yours
> > > >> > > Mikhail Khludnev
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Sincerely yours
> > > >> Mikhail Khludnev
> > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> >
>
--
Sincerely yours
Mikhail Khludnev
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Lee Carroll <le...@googlemail.com>.
Inventive. I need to control content-type of the response from the document
field value. I have the actual content field and the content-type field to
use configured in the response writer. I've just noticed that the xslt
transformer allows you to do this but not controlled by document values. I
may also need to set some headers based on content-type and perhaps content
size, accept ranges comes to mind. Although I might be getting ahead of
myself.
On 18 April 2018 at 12:05, Mikhail Khludnev <mk...@apache.org> wrote:
> well ..
> what if
> http://localhost:8983/solr/images/select?fl=content&q=id:1&start=1&wt=csv&
> csv.separator=&csv.encapsulator&csv.null=null
> ?
>
> On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll <lee.a.carroll@googlemail.com
> >
> wrote:
>
> > sorry cut n paste error i'd get
> >
> > {
> > "responseHeader":{
> > "zkConnected":true,
> > "status":0,
> > "QTime":0,
> > "params":{
> > "q":"*:*",
> > "fl":"content",
> > "rows":"1"}},
> > "response":{"numFound":1,"start":0,"docs":[
> > {
> > "content":"my-content-value"}]
> > }}
> >
> >
> > but you get my point
> >
> >
> >
> > On 18 April 2018 at 11:13, Lee Carroll <le...@googlemail.com>
> > wrote:
> >
> > > for http://localhost:8983/solr/images/select?fl=content&q=id:1&start=1
> > >
> > > I'd get
> > >
> > > {
> > > "responseHeader":{
> > > "zkConnected":true,
> > > "status":0,
> > > "QTime":1,
> > > "params":{
> > > "q":"*:*",
> > > "_":"1524046333220"}},
> > > "response":{"numFound":1,"start":0,"docs":[
> > > {
> > > "id":"1",
> > > "content":"my-content-value",
> > > "*content-type*":"text/plain"}]
> > > }}
> > >
> > > when i want
> > >
> > > my-content-value
> > >
> > >
> > >
> > > On 18 April 2018 at 10:55, Mikhail Khludnev <mk...@apache.org> wrote:
> > >
> > >> Lee, from this description I don see why it can't be addressed by
> > fl,rows
> > >> params. What makes it different form the typical Solr usage?
> > >>
> > >>
> > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll <
> > >> lee.a.carroll@googlemail.com>
> > >> wrote:
> > >>
> > >> > Sure, we want to return a single field's value for the top matching
> > >> > document for a given query. Bare content rather than a full search
> > >> result
> > >> > listing.
> > >> >
> > >> > To be concrete:
> > >> >
> > >> > For a schema of fields id [unique key],
> content[stored],content-type[
> > >> > stored]
> > >> > For a request:
> > >> >
> > >> > 1. Request URL:
> > >> > https://localhost/solr/content?q=id:1
> > >> > 2. Request Method:
> > >> > GET
> > >> >
> > >> > We get a response
> > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: [content-type
> > value]
> > >> >
> > >> > and the body to be the raw value of content
> > >> >
> > >> > In short clients consume directly the most relevant "content"
> returned
> > >> from
> > >> > solr queries they construct.
> > >> >
> > >> > Naively I've implemented a subclass of RawResponseWriter which takes
> > the
> > >> > first docs values and adds them to the appended "content" stream.
> > >> Should I
> > >> > selectively add the content stream depending on if this is the final
> > >> > aggregation of cloud results (and provide a base class writer to act
> > if
> > >> > not), if so how do I know its the final aggregation. Or is adding
> the
> > >> > content stream within the response writer a bad idea. Should that be
> > >> being
> > >> > added to the response somewhere else?
> > >> >
> > >> > Failing all of the above is asking about response writer an X / Y
> > >> problem.
> > >> > Is their a better way to achieve the above. I'd looked at
> transforming
> > >> > response xml but that seemed not to offer a complete bare slate.
> > >> >
> > >> > Cheers Lee C
> > >> >
> > >> >
> > >> > On 17 April 2018 at 21:36, Mikhail Khludnev <mk...@apache.org>
> wrote:
> > >> >
> > >> > > In distributed search response writer is used twice
> > >> > > https://lucene.apache.org/solr/guide/7_1/distributed-
> requests.html
> > >> > > once slave node that's where response writer yields "json" content
> > >> and it
> > >> > > upset aggregator node which is expect only javabin.
> > >> > > I hardly can comment on rrw, it's probably used for responding
> > >> separate
> > >> > > files in distrib=false mode.
> > >> > > You can start from describing why you need to create own response
> > >> writer.
> > >> > >
> > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <
> > >> > lee.a.carroll@googlemail.com
> > >> > > >
> > >> > > wrote:
> > >> > >
> > >> > > > Ok. My expectation was the response writer would not be used
> until
> > >> the
> > >> > > > final serialization of the result. If my response writer breaks
> > the
> > >> > > > response writer contract, exactly the way rawResponseWriter does
> > and
> > >> > just
> > >> > > > out puts a filed value how does that work? Does
> rawResponseWriter
> > >> > support
> > >> > > > cloud mode?
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev <mk...@apache.org>
> > >> wrote:
> > >> > > >
> > >> > > > > That's what should happen.
> > >> > > > >
> > >> > > > > Expected mime type application/octet-stream but got
> > >> application/json.
> > >> > > > >
> > >> > > > > Distributed search coordinator expect to merge slave responses
> > in
> > >> > > javabin
> > >> > > > > format. But slave's wt indicated json.
> > >> > > > > As far as I know only javabin might be used to distributed
> > search
> > >> > > > > underneath. Coordinator itself might yield json.
> > >> > > > >
> > >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
> > >> > > > lee.a.carroll@googlemail.com
> > >> > > > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Sure
> > >> > > > > >
> > >> > > > > > with 1 shard 1 replica this request works fine
> > >> > > > > >
> > >> > > > > > 1. Request URL:
> > >> > > > > > http://localhost:8983/solr/images/image?q=id:1
> > >> > > > > > 2. Request Method:
> > >> > > > > > GET
> > >> > > > > > 3. Status Code:
> > >> > > > > > 200 OK
> > >> > > > > >
> > >> > > > > > logs are clean
> > >> > > > > >
> > >> > > > > > with 2 shards 2 replicas the same request fails and in the
> > logs
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2
> > r:core_node7
> > >> > > > > > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore;
> > >> > > > > > [images_shard2_replica_n4] webapp=/solr path=/image
> > >> > > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > >> > > > > > http://10.224.30.207:8983/solr/images_shard2_replica_n4/
> > >> > > > > > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/
> > >> > > > > > &rows=10&version=2&q=id:1&NOW=
> 1523971232039&isShard=true&wt=
> > >> > javabin}
> > >> > > > > > hits=0 status=0 QTime=0
> > >> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1
> > r:core_node3
> > >> > > > > > x:images_shard1_replica_n1] org.apache.solr.common.SolrExc
> > >> eption;
> > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
> > >> > > RemoteSolrException:
> > >> > > > > > Error
> > >> > > > > > from server at http://10.224.30.207:8983/
> > >> > > solr/images_shard2_replica_n4
> > >> > > > :
> > >> > > > > > Expected mime type application/octet-stream but got
> > >> > application/json.
> > >> > > > > > at
> > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
> > >> > > > > > executeMethod(HttpSolrClient.java:607)
> > >> > > > > > at
> > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > >> > > > > > HttpSolrClient.java:255)
> > >> > > > > > at
> > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > >> > > > > > HttpSolrClient.java:244)
> > >> > > > > > at
> > >> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > >> > > > > > doRequest(LBHttpSolrClient.java:483)
> > >> > > > > > at
> > >> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(
> > >> > > > > > LBHttpSolrClient.java:413)
> > >> > > > > > at
> > >> > > > > > org.apache.solr.handler.component.HttpShardHandlerFactory.
> > >> > > > > > makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
> > >> > > > > > at
> > >> > > > > > org.apache.solr.handler.component.HttpShardHandler.
> > >> > lambda$submit$0(
> > >> > > > > > HttpShardHandler.java:175)
> > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > >> > > > > > at java.util.concurrent.Executors$RunnableAdapter.
> > >> > > > > call(Executors.java:511)
> > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > >> > > > > > at
> > >> > > > > > com.codahale.metrics.InstrumentedExecutorService$
> > >> > > > > InstrumentedRunnable.run(
> > >> > > > > > InstrumentedExecutorService.java:176)
> > >> > > > > > at
> > >> > > > > > org.apache.solr.common.util.ExecutorUtil$
> > >> > MDCAwareThreadPoolExecutor.
> > >> > > > > > lambda$execute$0(ExecutorUtil.java:188)
> > >> > > > > > at
> > >> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > >> > > > > > ThreadPoolExecutor.java:1142)
> > >> > > > > > at
> > >> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > >> > > > > > ThreadPoolExecutor.java:617)
> > >> > > > > > at java.lang.Thread.run(Thread.java:745)
> > >> > > > > >
> > >> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1
> > r:core_node3
> > >> > > > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> > >> > params={q=id:1}
> > >> > > > > > status=200 QTime=17
> > >> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1
> > r:core_node3
> > >> > > > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> > >> > > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > >> > > > > > http://10.224.30.207:8983/solr/images_shard1_replica_n1/
> > >> > > > > > |http://10.224.30.207:7574/solr/images_shard1_replica_n2/
> > >> > > > > > &rows=10&version=2&q=id:1&NOW=
> 1523971232039&isShard=true&wt=
> > >> > javabin}
> > >> > > > > > hits=1 status=0 QTime=2
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > I've implemented getcontenttype simply as
> > >> > > > > >
> > >> > > > > > @Override
> > >> > > > > > public String getContentType(SolrQueryRequest request,
> > >> > > > > > SolrQueryResponse response) {
> > >> > > > > >
> > >> > > > > > return "application/json;charset=utf-8";
> > >> > > > > > }
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev <
> mkhl@apache.org>
> > >> > wrote:
> > >> > > > > >
> > >> > > > > > > Lee,
> > >> > > > > > > It's worth to send a stacktrace for such kind of
> inquiries.
> > >> > > > > > > I guess it goes from QueryComponent.mergeIds() or so.
> Shard
> > >> > > response
> > >> > > > > > should
> > >> > > > > > > contains <uniqueKey> from schema.xml field.
> > >> > > > > > > I encounter something like this while troubleshooting
> > >> > > > > > > https://lucene.apache.org/solr/guide/6_6/transforming-
> > >> > > > > > > result-documents.html#TransformingResultDocuments-
> > >> > > > > > > CoresandCollectionsinSolrCloud
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
> > >> > > > > > lee.a.carroll@googlemail.com
> > >> > > > > > > >
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > I've created a custom response writer which extends
> > >> > > > > RawResponseWriter.
> > >> > > > > > > The
> > >> > > > > > > > basic operation is to output a single field value from
> the
> > >> top
> > >> > > > > matching
> > >> > > > > > > doc
> > >> > > > > > > > as the entire response. This works when shards = 1 but
> > fails
> > >> > when
> > >> > > > > > shards
> > >> > > > > > > > are greater than 1.
> > >> > > > > > > >
> > >> > > > > > > > I throw an error if the field in question is missing
> from
> > >> the
> > >> > top
> > >> > > > > doc.
> > >> > > > > > > This
> > >> > > > > > > > happens when individual shards are being searched and
> only
> > >> id
> > >> > and
> > >> > > > > score
> > >> > > > > > > are
> > >> > > > > > > > returned. I'm sure I've committed a basic error.
> > >> > > > > > > >
> > >> > > > > > > > Lee C
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > > Sincerely yours
> > >> > > > > > > Mikhail Khludnev
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > > Sincerely yours
> > >> > > > > Mikhail Khludnev
> > >> > > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Sincerely yours
> > >> > > Mikhail Khludnev
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Sincerely yours
> > >> Mikhail Khludnev
> > >>
> > >
> > >
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Mikhail Khludnev <mk...@apache.org>.
well ..
what if
http://localhost:8983/solr/images/select?fl=content&q=id:1&start=1&wt=csv&
csv.separator=&csv.encapsulator&csv.null=null
?
On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll <le...@googlemail.com>
wrote:
> sorry cut n paste error i'd get
>
> {
> "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":0,
> "params":{
> "q":"*:*",
> "fl":"content",
> "rows":"1"}},
> "response":{"numFound":1,"start":0,"docs":[
> {
> "content":"my-content-value"}]
> }}
>
>
> but you get my point
>
>
>
> On 18 April 2018 at 11:13, Lee Carroll <le...@googlemail.com>
> wrote:
>
> > for http://localhost:8983/solr/images/select?fl=content&q=id:1&start=1
> >
> > I'd get
> >
> > {
> > "responseHeader":{
> > "zkConnected":true,
> > "status":0,
> > "QTime":1,
> > "params":{
> > "q":"*:*",
> > "_":"1524046333220"}},
> > "response":{"numFound":1,"start":0,"docs":[
> > {
> > "id":"1",
> > "content":"my-content-value",
> > "*content-type*":"text/plain"}]
> > }}
> >
> > when i want
> >
> > my-content-value
> >
> >
> >
> > On 18 April 2018 at 10:55, Mikhail Khludnev <mk...@apache.org> wrote:
> >
> >> Lee, from this description I don see why it can't be addressed by
> fl,rows
> >> params. What makes it different form the typical Solr usage?
> >>
> >>
> >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll <
> >> lee.a.carroll@googlemail.com>
> >> wrote:
> >>
> >> > Sure, we want to return a single field's value for the top matching
> >> > document for a given query. Bare content rather than a full search
> >> result
> >> > listing.
> >> >
> >> > To be concrete:
> >> >
> >> > For a schema of fields id [unique key], content[stored],content-type[
> >> > stored]
> >> > For a request:
> >> >
> >> > 1. Request URL:
> >> > https://localhost/solr/content?q=id:1
> >> > 2. Request Method:
> >> > GET
> >> >
> >> > We get a response
> >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: [content-type
> value]
> >> >
> >> > and the body to be the raw value of content
> >> >
> >> > In short clients consume directly the most relevant "content" returned
> >> from
> >> > solr queries they construct.
> >> >
> >> > Naively I've implemented a subclass of RawResponseWriter which takes
> the
> >> > first docs values and adds them to the appended "content" stream.
> >> Should I
> >> > selectively add the content stream depending on if this is the final
> >> > aggregation of cloud results (and provide a base class writer to act
> if
> >> > not), if so how do I know its the final aggregation. Or is adding the
> >> > content stream within the response writer a bad idea. Should that be
> >> being
> >> > added to the response somewhere else?
> >> >
> >> > Failing all of the above is asking about response writer an X / Y
> >> problem.
> >> > Is their a better way to achieve the above. I'd looked at transforming
> >> > response xml but that seemed not to offer a complete bare slate.
> >> >
> >> > Cheers Lee C
> >> >
> >> >
> >> > On 17 April 2018 at 21:36, Mikhail Khludnev <mk...@apache.org> wrote:
> >> >
> >> > > In distributed search response writer is used twice
> >> > > https://lucene.apache.org/solr/guide/7_1/distributed-requests.html
> >> > > once slave node that's where response writer yields "json" content
> >> and it
> >> > > upset aggregator node which is expect only javabin.
> >> > > I hardly can comment on rrw, it's probably used for responding
> >> separate
> >> > > files in distrib=false mode.
> >> > > You can start from describing why you need to create own response
> >> writer.
> >> > >
> >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <
> >> > lee.a.carroll@googlemail.com
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > Ok. My expectation was the response writer would not be used until
> >> the
> >> > > > final serialization of the result. If my response writer breaks
> the
> >> > > > response writer contract, exactly the way rawResponseWriter does
> and
> >> > just
> >> > > > out puts a filed value how does that work? Does rawResponseWriter
> >> > support
> >> > > > cloud mode?
> >> > > >
> >> > > >
> >> > > >
> >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev <mk...@apache.org>
> >> wrote:
> >> > > >
> >> > > > > That's what should happen.
> >> > > > >
> >> > > > > Expected mime type application/octet-stream but got
> >> application/json.
> >> > > > >
> >> > > > > Distributed search coordinator expect to merge slave responses
> in
> >> > > javabin
> >> > > > > format. But slave's wt indicated json.
> >> > > > > As far as I know only javabin might be used to distributed
> search
> >> > > > > underneath. Coordinator itself might yield json.
> >> > > > >
> >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
> >> > > > lee.a.carroll@googlemail.com
> >> > > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Sure
> >> > > > > >
> >> > > > > > with 1 shard 1 replica this request works fine
> >> > > > > >
> >> > > > > > 1. Request URL:
> >> > > > > > http://localhost:8983/solr/images/image?q=id:1
> >> > > > > > 2. Request Method:
> >> > > > > > GET
> >> > > > > > 3. Status Code:
> >> > > > > > 200 OK
> >> > > > > >
> >> > > > > > logs are clean
> >> > > > > >
> >> > > > > > with 2 shards 2 replicas the same request fails and in the
> logs
> >> > > > > >
> >> > > > > >
> >> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2
> r:core_node7
> >> > > > > > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore;
> >> > > > > > [images_shard2_replica_n4] webapp=/solr path=/image
> >> > > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> >> > > > > > http://10.224.30.207:8983/solr/images_shard2_replica_n4/
> >> > > > > > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/
> >> > > > > > &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=
> >> > javabin}
> >> > > > > > hits=0 status=0 QTime=0
> >> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1
> r:core_node3
> >> > > > > > x:images_shard1_replica_n1] org.apache.solr.common.SolrExc
> >> eption;
> >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
> >> > > RemoteSolrException:
> >> > > > > > Error
> >> > > > > > from server at http://10.224.30.207:8983/
> >> > > solr/images_shard2_replica_n4
> >> > > > :
> >> > > > > > Expected mime type application/octet-stream but got
> >> > application/json.
> >> > > > > > at
> >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
> >> > > > > > executeMethod(HttpSolrClient.java:607)
> >> > > > > > at
> >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> >> > > > > > HttpSolrClient.java:255)
> >> > > > > > at
> >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> >> > > > > > HttpSolrClient.java:244)
> >> > > > > > at
> >> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> >> > > > > > doRequest(LBHttpSolrClient.java:483)
> >> > > > > > at
> >> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(
> >> > > > > > LBHttpSolrClient.java:413)
> >> > > > > > at
> >> > > > > > org.apache.solr.handler.component.HttpShardHandlerFactory.
> >> > > > > > makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
> >> > > > > > at
> >> > > > > > org.apache.solr.handler.component.HttpShardHandler.
> >> > lambda$submit$0(
> >> > > > > > HttpShardHandler.java:175)
> >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> > > > > > at java.util.concurrent.Executors$RunnableAdapter.
> >> > > > > call(Executors.java:511)
> >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> > > > > > at
> >> > > > > > com.codahale.metrics.InstrumentedExecutorService$
> >> > > > > InstrumentedRunnable.run(
> >> > > > > > InstrumentedExecutorService.java:176)
> >> > > > > > at
> >> > > > > > org.apache.solr.common.util.ExecutorUtil$
> >> > MDCAwareThreadPoolExecutor.
> >> > > > > > lambda$execute$0(ExecutorUtil.java:188)
> >> > > > > > at
> >> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> >> > > > > > ThreadPoolExecutor.java:1142)
> >> > > > > > at
> >> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> >> > > > > > ThreadPoolExecutor.java:617)
> >> > > > > > at java.lang.Thread.run(Thread.java:745)
> >> > > > > >
> >> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1
> r:core_node3
> >> > > > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> >> > params={q=id:1}
> >> > > > > > status=200 QTime=17
> >> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1
> r:core_node3
> >> > > > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> >> > > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> >> > > > > > http://10.224.30.207:8983/solr/images_shard1_replica_n1/
> >> > > > > > |http://10.224.30.207:7574/solr/images_shard1_replica_n2/
> >> > > > > > &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=
> >> > javabin}
> >> > > > > > hits=1 status=0 QTime=2
> >> > > > > >
> >> > > > > >
> >> > > > > > I've implemented getcontenttype simply as
> >> > > > > >
> >> > > > > > @Override
> >> > > > > > public String getContentType(SolrQueryRequest request,
> >> > > > > > SolrQueryResponse response) {
> >> > > > > >
> >> > > > > > return "application/json;charset=utf-8";
> >> > > > > > }
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev <mk...@apache.org>
> >> > wrote:
> >> > > > > >
> >> > > > > > > Lee,
> >> > > > > > > It's worth to send a stacktrace for such kind of inquiries.
> >> > > > > > > I guess it goes from QueryComponent.mergeIds() or so. Shard
> >> > > response
> >> > > > > > should
> >> > > > > > > contains <uniqueKey> from schema.xml field.
> >> > > > > > > I encounter something like this while troubleshooting
> >> > > > > > > https://lucene.apache.org/solr/guide/6_6/transforming-
> >> > > > > > > result-documents.html#TransformingResultDocuments-
> >> > > > > > > CoresandCollectionsinSolrCloud
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
> >> > > > > > lee.a.carroll@googlemail.com
> >> > > > > > > >
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > I've created a custom response writer which extends
> >> > > > > RawResponseWriter.
> >> > > > > > > The
> >> > > > > > > > basic operation is to output a single field value from the
> >> top
> >> > > > > matching
> >> > > > > > > doc
> >> > > > > > > > as the entire response. This works when shards = 1 but
> fails
> >> > when
> >> > > > > > shards
> >> > > > > > > > are greater than 1.
> >> > > > > > > >
> >> > > > > > > > I throw an error if the field in question is missing from
> >> the
> >> > top
> >> > > > > doc.
> >> > > > > > > This
> >> > > > > > > > happens when individual shards are being searched and only
> >> id
> >> > and
> >> > > > > score
> >> > > > > > > are
> >> > > > > > > > returned. I'm sure I've committed a basic error.
> >> > > > > > > >
> >> > > > > > > > Lee C
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > Sincerely yours
> >> > > > > > > Mikhail Khludnev
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Sincerely yours
> >> > > > > Mikhail Khludnev
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Sincerely yours
> >> > > Mikhail Khludnev
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Sincerely yours
> >> Mikhail Khludnev
> >>
> >
> >
>
--
Sincerely yours
Mikhail Khludnev
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Lee Carroll <le...@googlemail.com>.
sorry cut n paste error i'd get
{
"responseHeader":{
"zkConnected":true,
"status":0,
"QTime":0,
"params":{
"q":"*:*",
"fl":"content",
"rows":"1"}},
"response":{"numFound":1,"start":0,"docs":[
{
"content":"my-content-value"}]
}}
but you get my point
On 18 April 2018 at 11:13, Lee Carroll <le...@googlemail.com> wrote:
> for http://localhost:8983/solr/images/select?fl=content&q=id:1&start=1
>
> I'd get
>
> {
> "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":1,
> "params":{
> "q":"*:*",
> "_":"1524046333220"}},
> "response":{"numFound":1,"start":0,"docs":[
> {
> "id":"1",
> "content":"my-content-value",
> "*content-type*":"text/plain"}]
> }}
>
> when i want
>
> my-content-value
>
>
>
> On 18 April 2018 at 10:55, Mikhail Khludnev <mk...@apache.org> wrote:
>
>> Lee, from this description I don see why it can't be addressed by fl,rows
>> params. What makes it different form the typical Solr usage?
>>
>>
>> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll <
>> lee.a.carroll@googlemail.com>
>> wrote:
>>
>> > Sure, we want to return a single field's value for the top matching
>> > document for a given query. Bare content rather than a full search
>> result
>> > listing.
>> >
>> > To be concrete:
>> >
>> > For a schema of fields id [unique key], content[stored],content-type[
>> > stored]
>> > For a request:
>> >
>> > 1. Request URL:
>> > https://localhost/solr/content?q=id:1
>> > 2. Request Method:
>> > GET
>> >
>> > We get a response
>> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: [content-type value]
>> >
>> > and the body to be the raw value of content
>> >
>> > In short clients consume directly the most relevant "content" returned
>> from
>> > solr queries they construct.
>> >
>> > Naively I've implemented a subclass of RawResponseWriter which takes the
>> > first docs values and adds them to the appended "content" stream.
>> Should I
>> > selectively add the content stream depending on if this is the final
>> > aggregation of cloud results (and provide a base class writer to act if
>> > not), if so how do I know its the final aggregation. Or is adding the
>> > content stream within the response writer a bad idea. Should that be
>> being
>> > added to the response somewhere else?
>> >
>> > Failing all of the above is asking about response writer an X / Y
>> problem.
>> > Is their a better way to achieve the above. I'd looked at transforming
>> > response xml but that seemed not to offer a complete bare slate.
>> >
>> > Cheers Lee C
>> >
>> >
>> > On 17 April 2018 at 21:36, Mikhail Khludnev <mk...@apache.org> wrote:
>> >
>> > > In distributed search response writer is used twice
>> > > https://lucene.apache.org/solr/guide/7_1/distributed-requests.html
>> > > once slave node that's where response writer yields "json" content
>> and it
>> > > upset aggregator node which is expect only javabin.
>> > > I hardly can comment on rrw, it's probably used for responding
>> separate
>> > > files in distrib=false mode.
>> > > You can start from describing why you need to create own response
>> writer.
>> > >
>> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <
>> > lee.a.carroll@googlemail.com
>> > > >
>> > > wrote:
>> > >
>> > > > Ok. My expectation was the response writer would not be used until
>> the
>> > > > final serialization of the result. If my response writer breaks the
>> > > > response writer contract, exactly the way rawResponseWriter does and
>> > just
>> > > > out puts a filed value how does that work? Does rawResponseWriter
>> > support
>> > > > cloud mode?
>> > > >
>> > > >
>> > > >
>> > > > On 17 April 2018 at 15:55, Mikhail Khludnev <mk...@apache.org>
>> wrote:
>> > > >
>> > > > > That's what should happen.
>> > > > >
>> > > > > Expected mime type application/octet-stream but got
>> application/json.
>> > > > >
>> > > > > Distributed search coordinator expect to merge slave responses in
>> > > javabin
>> > > > > format. But slave's wt indicated json.
>> > > > > As far as I know only javabin might be used to distributed search
>> > > > > underneath. Coordinator itself might yield json.
>> > > > >
>> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
>> > > > lee.a.carroll@googlemail.com
>> > > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Sure
>> > > > > >
>> > > > > > with 1 shard 1 replica this request works fine
>> > > > > >
>> > > > > > 1. Request URL:
>> > > > > > http://localhost:8983/solr/images/image?q=id:1
>> > > > > > 2. Request Method:
>> > > > > > GET
>> > > > > > 3. Status Code:
>> > > > > > 200 OK
>> > > > > >
>> > > > > > logs are clean
>> > > > > >
>> > > > > > with 2 shards 2 replicas the same request fails and in the logs
>> > > > > >
>> > > > > >
>> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7
>> > > > > > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore;
>> > > > > > [images_shard2_replica_n4] webapp=/solr path=/image
>> > > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
>> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
>> > > > > > http://10.224.30.207:8983/solr/images_shard2_replica_n4/
>> > > > > > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/
>> > > > > > &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=
>> > javabin}
>> > > > > > hits=0 status=0 QTime=0
>> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
>> > > > > > x:images_shard1_replica_n1] org.apache.solr.common.SolrExc
>> eption;
>> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
>> > > RemoteSolrException:
>> > > > > > Error
>> > > > > > from server at http://10.224.30.207:8983/
>> > > solr/images_shard2_replica_n4
>> > > > :
>> > > > > > Expected mime type application/octet-stream but got
>> > application/json.
>> > > > > > at
>> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
>> > > > > > executeMethod(HttpSolrClient.java:607)
>> > > > > > at
>> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
>> > > > > > HttpSolrClient.java:255)
>> > > > > > at
>> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
>> > > > > > HttpSolrClient.java:244)
>> > > > > > at
>> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.
>> > > > > > doRequest(LBHttpSolrClient.java:483)
>> > > > > > at
>> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(
>> > > > > > LBHttpSolrClient.java:413)
>> > > > > > at
>> > > > > > org.apache.solr.handler.component.HttpShardHandlerFactory.
>> > > > > > makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
>> > > > > > at
>> > > > > > org.apache.solr.handler.component.HttpShardHandler.
>> > lambda$submit$0(
>> > > > > > HttpShardHandler.java:175)
>> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > > > > > at java.util.concurrent.Executors$RunnableAdapter.
>> > > > > call(Executors.java:511)
>> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > > > > > at
>> > > > > > com.codahale.metrics.InstrumentedExecutorService$
>> > > > > InstrumentedRunnable.run(
>> > > > > > InstrumentedExecutorService.java:176)
>> > > > > > at
>> > > > > > org.apache.solr.common.util.ExecutorUtil$
>> > MDCAwareThreadPoolExecutor.
>> > > > > > lambda$execute$0(ExecutorUtil.java:188)
>> > > > > > at
>> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
>> > > > > > ThreadPoolExecutor.java:1142)
>> > > > > > at
>> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> > > > > > ThreadPoolExecutor.java:617)
>> > > > > > at java.lang.Thread.run(Thread.java:745)
>> > > > > >
>> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1 r:core_node3
>> > > > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
>> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
>> > params={q=id:1}
>> > > > > > status=200 QTime=17
>> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
>> > > > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
>> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
>> > > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
>> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
>> > > > > > http://10.224.30.207:8983/solr/images_shard1_replica_n1/
>> > > > > > |http://10.224.30.207:7574/solr/images_shard1_replica_n2/
>> > > > > > &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=
>> > javabin}
>> > > > > > hits=1 status=0 QTime=2
>> > > > > >
>> > > > > >
>> > > > > > I've implemented getcontenttype simply as
>> > > > > >
>> > > > > > @Override
>> > > > > > public String getContentType(SolrQueryRequest request,
>> > > > > > SolrQueryResponse response) {
>> > > > > >
>> > > > > > return "application/json;charset=utf-8";
>> > > > > > }
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev <mk...@apache.org>
>> > wrote:
>> > > > > >
>> > > > > > > Lee,
>> > > > > > > It's worth to send a stacktrace for such kind of inquiries.
>> > > > > > > I guess it goes from QueryComponent.mergeIds() or so. Shard
>> > > response
>> > > > > > should
>> > > > > > > contains <uniqueKey> from schema.xml field.
>> > > > > > > I encounter something like this while troubleshooting
>> > > > > > > https://lucene.apache.org/solr/guide/6_6/transforming-
>> > > > > > > result-documents.html#TransformingResultDocuments-
>> > > > > > > CoresandCollectionsinSolrCloud
>> > > > > > >
>> > > > > > >
>> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
>> > > > > > lee.a.carroll@googlemail.com
>> > > > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > I've created a custom response writer which extends
>> > > > > RawResponseWriter.
>> > > > > > > The
>> > > > > > > > basic operation is to output a single field value from the
>> top
>> > > > > matching
>> > > > > > > doc
>> > > > > > > > as the entire response. This works when shards = 1 but fails
>> > when
>> > > > > > shards
>> > > > > > > > are greater than 1.
>> > > > > > > >
>> > > > > > > > I throw an error if the field in question is missing from
>> the
>> > top
>> > > > > doc.
>> > > > > > > This
>> > > > > > > > happens when individual shards are being searched and only
>> id
>> > and
>> > > > > score
>> > > > > > > are
>> > > > > > > > returned. I'm sure I've committed a basic error.
>> > > > > > > >
>> > > > > > > > Lee C
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Sincerely yours
>> > > > > > > Mikhail Khludnev
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Sincerely yours
>> > > > > Mikhail Khludnev
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Sincerely yours
>> > > Mikhail Khludnev
>> > >
>> >
>>
>>
>>
>> --
>> Sincerely yours
>> Mikhail Khludnev
>>
>
>
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Lee Carroll <le...@googlemail.com>.
for http://localhost:8983/solr/images/select?fl=content&q=id:1&start=1
I'd get
{
"responseHeader":{
"zkConnected":true,
"status":0,
"QTime":1,
"params":{
"q":"*:*",
"_":"1524046333220"}},
"response":{"numFound":1,"start":0,"docs":[
{
"id":"1",
"content":"my-content-value",
"*content-type*":"text/plain"}]
}}
when i want
my-content-value
On 18 April 2018 at 10:55, Mikhail Khludnev <mk...@apache.org> wrote:
> Lee, from this description I don see why it can't be addressed by fl,rows
> params. What makes it different form the typical Solr usage?
>
>
> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll <
> lee.a.carroll@googlemail.com>
> wrote:
>
> > Sure, we want to return a single field's value for the top matching
> > document for a given query. Bare content rather than a full search result
> > listing.
> >
> > To be concrete:
> >
> > For a schema of fields id [unique key], content[stored],content-type[
> > stored]
> > For a request:
> >
> > 1. Request URL:
> > https://localhost/solr/content?q=id:1
> > 2. Request Method:
> > GET
> >
> > We get a response
> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: [content-type value]
> >
> > and the body to be the raw value of content
> >
> > In short clients consume directly the most relevant "content" returned
> from
> > solr queries they construct.
> >
> > Naively I've implemented a subclass of RawResponseWriter which takes the
> > first docs values and adds them to the appended "content" stream. Should
> I
> > selectively add the content stream depending on if this is the final
> > aggregation of cloud results (and provide a base class writer to act if
> > not), if so how do I know its the final aggregation. Or is adding the
> > content stream within the response writer a bad idea. Should that be
> being
> > added to the response somewhere else?
> >
> > Failing all of the above is asking about response writer an X / Y
> problem.
> > Is their a better way to achieve the above. I'd looked at transforming
> > response xml but that seemed not to offer a complete bare slate.
> >
> > Cheers Lee C
> >
> >
> > On 17 April 2018 at 21:36, Mikhail Khludnev <mk...@apache.org> wrote:
> >
> > > In distributed search response writer is used twice
> > > https://lucene.apache.org/solr/guide/7_1/distributed-requests.html
> > > once slave node that's where response writer yields "json" content and
> it
> > > upset aggregator node which is expect only javabin.
> > > I hardly can comment on rrw, it's probably used for responding separate
> > > files in distrib=false mode.
> > > You can start from describing why you need to create own response
> writer.
> > >
> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <
> > lee.a.carroll@googlemail.com
> > > >
> > > wrote:
> > >
> > > > Ok. My expectation was the response writer would not be used until
> the
> > > > final serialization of the result. If my response writer breaks the
> > > > response writer contract, exactly the way rawResponseWriter does and
> > just
> > > > out puts a filed value how does that work? Does rawResponseWriter
> > support
> > > > cloud mode?
> > > >
> > > >
> > > >
> > > > On 17 April 2018 at 15:55, Mikhail Khludnev <mk...@apache.org> wrote:
> > > >
> > > > > That's what should happen.
> > > > >
> > > > > Expected mime type application/octet-stream but got
> application/json.
> > > > >
> > > > > Distributed search coordinator expect to merge slave responses in
> > > javabin
> > > > > format. But slave's wt indicated json.
> > > > > As far as I know only javabin might be used to distributed search
> > > > > underneath. Coordinator itself might yield json.
> > > > >
> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
> > > > lee.a.carroll@googlemail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Sure
> > > > > >
> > > > > > with 1 shard 1 replica this request works fine
> > > > > >
> > > > > > 1. Request URL:
> > > > > > http://localhost:8983/solr/images/image?q=id:1
> > > > > > 2. Request Method:
> > > > > > GET
> > > > > > 3. Status Code:
> > > > > > 200 OK
> > > > > >
> > > > > > logs are clean
> > > > > >
> > > > > > with 2 shards 2 replicas the same request fails and in the logs
> > > > > >
> > > > > >
> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7
> > > > > > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore;
> > > > > > [images_shard2_replica_n4] webapp=/solr path=/image
> > > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > > > > http://10.224.30.207:8983/solr/images_shard2_replica_n4/
> > > > > > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/
> > > > > > &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=
> > javabin}
> > > > > > hits=0 status=0 QTime=0
> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
> > > > > > x:images_shard1_replica_n1] org.apache.solr.common.
> SolrException;
> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
> > > RemoteSolrException:
> > > > > > Error
> > > > > > from server at http://10.224.30.207:8983/
> > > solr/images_shard2_replica_n4
> > > > :
> > > > > > Expected mime type application/octet-stream but got
> > application/json.
> > > > > > at
> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
> > > > > > executeMethod(HttpSolrClient.java:607)
> > > > > > at
> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > > > > > HttpSolrClient.java:255)
> > > > > > at
> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > > > > > HttpSolrClient.java:244)
> > > > > > at
> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > > > > > doRequest(LBHttpSolrClient.java:483)
> > > > > > at
> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(
> > > > > > LBHttpSolrClient.java:413)
> > > > > > at
> > > > > > org.apache.solr.handler.component.HttpShardHandlerFactory.
> > > > > > makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
> > > > > > at
> > > > > > org.apache.solr.handler.component.HttpShardHandler.
> > lambda$submit$0(
> > > > > > HttpShardHandler.java:175)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > at java.util.concurrent.Executors$RunnableAdapter.
> > > > > call(Executors.java:511)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > at
> > > > > > com.codahale.metrics.InstrumentedExecutorService$
> > > > > InstrumentedRunnable.run(
> > > > > > InstrumentedExecutorService.java:176)
> > > > > > at
> > > > > > org.apache.solr.common.util.ExecutorUtil$
> > MDCAwareThreadPoolExecutor.
> > > > > > lambda$execute$0(ExecutorUtil.java:188)
> > > > > > at
> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > ThreadPoolExecutor.java:1142)
> > > > > > at
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > ThreadPoolExecutor.java:617)
> > > > > > at java.lang.Thread.run(Thread.java:745)
> > > > > >
> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1 r:core_node3
> > > > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> > params={q=id:1}
> > > > > > status=200 QTime=17
> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
> > > > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> > > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > > > > http://10.224.30.207:8983/solr/images_shard1_replica_n1/
> > > > > > |http://10.224.30.207:7574/solr/images_shard1_replica_n2/
> > > > > > &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=
> > javabin}
> > > > > > hits=1 status=0 QTime=2
> > > > > >
> > > > > >
> > > > > > I've implemented getcontenttype simply as
> > > > > >
> > > > > > @Override
> > > > > > public String getContentType(SolrQueryRequest request,
> > > > > > SolrQueryResponse response) {
> > > > > >
> > > > > > return "application/json;charset=utf-8";
> > > > > > }
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev <mk...@apache.org>
> > wrote:
> > > > > >
> > > > > > > Lee,
> > > > > > > It's worth to send a stacktrace for such kind of inquiries.
> > > > > > > I guess it goes from QueryComponent.mergeIds() or so. Shard
> > > response
> > > > > > should
> > > > > > > contains <uniqueKey> from schema.xml field.
> > > > > > > I encounter something like this while troubleshooting
> > > > > > > https://lucene.apache.org/solr/guide/6_6/transforming-
> > > > > > > result-documents.html#TransformingResultDocuments-
> > > > > > > CoresandCollectionsinSolrCloud
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
> > > > > > lee.a.carroll@googlemail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I've created a custom response writer which extends
> > > > > RawResponseWriter.
> > > > > > > The
> > > > > > > > basic operation is to output a single field value from the
> top
> > > > > matching
> > > > > > > doc
> > > > > > > > as the entire response. This works when shards = 1 but fails
> > when
> > > > > > shards
> > > > > > > > are greater than 1.
> > > > > > > >
> > > > > > > > I throw an error if the field in question is missing from the
> > top
> > > > > doc.
> > > > > > > This
> > > > > > > > happens when individual shards are being searched and only id
> > and
> > > > > score
> > > > > > > are
> > > > > > > > returned. I'm sure I've committed a basic error.
> > > > > > > >
> > > > > > > > Lee C
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Sincerely yours
> > > > > > > Mikhail Khludnev
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sincerely yours
> > > > > Mikhail Khludnev
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > >
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Mikhail Khludnev <mk...@apache.org>.
Lee, from this description I don see why it can't be addressed by fl,rows
params. What makes it different form the typical Solr usage?
On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll <le...@googlemail.com>
wrote:
> Sure, we want to return a single field's value for the top matching
> document for a given query. Bare content rather than a full search result
> listing.
>
> To be concrete:
>
> For a schema of fields id [unique key], content[stored],content-type[
> stored]
> For a request:
>
> 1. Request URL:
> https://localhost/solr/content?q=id:1
> 2. Request Method:
> GET
>
> We get a response
> HTTP/1.1 200 OK Content-Length: 16261 Content-Type: [content-type value]
>
> and the body to be the raw value of content
>
> In short clients consume directly the most relevant "content" returned from
> solr queries they construct.
>
> Naively I've implemented a subclass of RawResponseWriter which takes the
> first docs values and adds them to the appended "content" stream. Should I
> selectively add the content stream depending on if this is the final
> aggregation of cloud results (and provide a base class writer to act if
> not), if so how do I know its the final aggregation. Or is adding the
> content stream within the response writer a bad idea. Should that be being
> added to the response somewhere else?
>
> Failing all of the above is asking about response writer an X / Y problem.
> Is their a better way to achieve the above. I'd looked at transforming
> response xml but that seemed not to offer a complete bare slate.
>
> Cheers Lee C
>
>
> On 17 April 2018 at 21:36, Mikhail Khludnev <mk...@apache.org> wrote:
>
> > In distributed search response writer is used twice
> > https://lucene.apache.org/solr/guide/7_1/distributed-requests.html
> > once slave node that's where response writer yields "json" content and it
> > upset aggregator node which is expect only javabin.
> > I hardly can comment on rrw, it's probably used for responding separate
> > files in distrib=false mode.
> > You can start from describing why you need to create own response writer.
> >
> > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <
> lee.a.carroll@googlemail.com
> > >
> > wrote:
> >
> > > Ok. My expectation was the response writer would not be used until the
> > > final serialization of the result. If my response writer breaks the
> > > response writer contract, exactly the way rawResponseWriter does and
> just
> > > out puts a filed value how does that work? Does rawResponseWriter
> support
> > > cloud mode?
> > >
> > >
> > >
> > > On 17 April 2018 at 15:55, Mikhail Khludnev <mk...@apache.org> wrote:
> > >
> > > > That's what should happen.
> > > >
> > > > Expected mime type application/octet-stream but got application/json.
> > > >
> > > > Distributed search coordinator expect to merge slave responses in
> > javabin
> > > > format. But slave's wt indicated json.
> > > > As far as I know only javabin might be used to distributed search
> > > > underneath. Coordinator itself might yield json.
> > > >
> > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
> > > lee.a.carroll@googlemail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Sure
> > > > >
> > > > > with 1 shard 1 replica this request works fine
> > > > >
> > > > > 1. Request URL:
> > > > > http://localhost:8983/solr/images/image?q=id:1
> > > > > 2. Request Method:
> > > > > GET
> > > > > 3. Status Code:
> > > > > 200 OK
> > > > >
> > > > > logs are clean
> > > > >
> > > > > with 2 shards 2 replicas the same request fails and in the logs
> > > > >
> > > > >
> > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7
> > > > > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore;
> > > > > [images_shard2_replica_n4] webapp=/solr path=/image
> > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > > > http://10.224.30.207:8983/solr/images_shard2_replica_n4/
> > > > > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/
> > > > > &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=
> javabin}
> > > > > hits=0 status=0 QTime=0
> > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
> > > > > x:images_shard1_replica_n1] org.apache.solr.common.SolrException;
> > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
> > RemoteSolrException:
> > > > > Error
> > > > > from server at http://10.224.30.207:8983/
> > solr/images_shard2_replica_n4
> > > :
> > > > > Expected mime type application/octet-stream but got
> application/json.
> > > > > at
> > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
> > > > > executeMethod(HttpSolrClient.java:607)
> > > > > at
> > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > > > > HttpSolrClient.java:255)
> > > > > at
> > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > > > > HttpSolrClient.java:244)
> > > > > at
> > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > > > > doRequest(LBHttpSolrClient.java:483)
> > > > > at
> > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(
> > > > > LBHttpSolrClient.java:413)
> > > > > at
> > > > > org.apache.solr.handler.component.HttpShardHandlerFactory.
> > > > > makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
> > > > > at
> > > > > org.apache.solr.handler.component.HttpShardHandler.
> lambda$submit$0(
> > > > > HttpShardHandler.java:175)
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > at java.util.concurrent.Executors$RunnableAdapter.
> > > > call(Executors.java:511)
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > at
> > > > > com.codahale.metrics.InstrumentedExecutorService$
> > > > InstrumentedRunnable.run(
> > > > > InstrumentedExecutorService.java:176)
> > > > > at
> > > > > org.apache.solr.common.util.ExecutorUtil$
> MDCAwareThreadPoolExecutor.
> > > > > lambda$execute$0(ExecutorUtil.java:188)
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > ThreadPoolExecutor.java:1142)
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > ThreadPoolExecutor.java:617)
> > > > > at java.lang.Thread.run(Thread.java:745)
> > > > >
> > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1 r:core_node3
> > > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> params={q=id:1}
> > > > > status=200 QTime=17
> > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
> > > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> > > > > [images_shard1_replica_n1] webapp=/solr path=/image
> > > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > > > http://10.224.30.207:8983/solr/images_shard1_replica_n1/
> > > > > |http://10.224.30.207:7574/solr/images_shard1_replica_n2/
> > > > > &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=
> javabin}
> > > > > hits=1 status=0 QTime=2
> > > > >
> > > > >
> > > > > I've implemented getcontenttype simply as
> > > > >
> > > > > @Override
> > > > > public String getContentType(SolrQueryRequest request,
> > > > > SolrQueryResponse response) {
> > > > >
> > > > > return "application/json;charset=utf-8";
> > > > > }
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 16 April 2018 at 17:37, Mikhail Khludnev <mk...@apache.org>
> wrote:
> > > > >
> > > > > > Lee,
> > > > > > It's worth to send a stacktrace for such kind of inquiries.
> > > > > > I guess it goes from QueryComponent.mergeIds() or so. Shard
> > response
> > > > > should
> > > > > > contains <uniqueKey> from schema.xml field.
> > > > > > I encounter something like this while troubleshooting
> > > > > > https://lucene.apache.org/solr/guide/6_6/transforming-
> > > > > > result-documents.html#TransformingResultDocuments-
> > > > > > CoresandCollectionsinSolrCloud
> > > > > >
> > > > > >
> > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
> > > > > lee.a.carroll@googlemail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I've created a custom response writer which extends
> > > > RawResponseWriter.
> > > > > > The
> > > > > > > basic operation is to output a single field value from the top
> > > > matching
> > > > > > doc
> > > > > > > as the entire response. This works when shards = 1 but fails
> when
> > > > > shards
> > > > > > > are greater than 1.
> > > > > > >
> > > > > > > I throw an error if the field in question is missing from the
> top
> > > > doc.
> > > > > > This
> > > > > > > happens when individual shards are being searched and only id
> and
> > > > score
> > > > > > are
> > > > > > > returned. I'm sure I've committed a basic error.
> > > > > > >
> > > > > > > Lee C
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Sincerely yours
> > > > > > Mikhail Khludnev
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sincerely yours
> > > > Mikhail Khludnev
> > > >
> > >
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> >
>
--
Sincerely yours
Mikhail Khludnev
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Lee Carroll <le...@googlemail.com>.
Sure, we want to return a single field's value for the top matching
document for a given query. Bare content rather than a full search result
listing.
To be concrete:
For a schema of fields id [unique key], content[stored],content-type[stored]
For a request:
1. Request URL:
https://localhost/solr/content?q=id:1
2. Request Method:
GET
We get a response
HTTP/1.1 200 OK Content-Length: 16261 Content-Type: [content-type value]
and the body to be the raw value of content
In short clients consume directly the most relevant "content" returned from
solr queries they construct.
Naively I've implemented a subclass of RawResponseWriter which takes the
first docs values and adds them to the appended "content" stream. Should I
selectively add the content stream depending on if this is the final
aggregation of cloud results (and provide a base class writer to act if
not), if so how do I know its the final aggregation. Or is adding the
content stream within the response writer a bad idea. Should that be being
added to the response somewhere else?
Failing all of the above is asking about response writer an X / Y problem.
Is their a better way to achieve the above. I'd looked at transforming
response xml but that seemed not to offer a complete bare slate.
Cheers Lee C
On 17 April 2018 at 21:36, Mikhail Khludnev <mk...@apache.org> wrote:
> In distributed search response writer is used twice
> https://lucene.apache.org/solr/guide/7_1/distributed-requests.html
> once slave node that's where response writer yields "json" content and it
> upset aggregator node which is expect only javabin.
> I hardly can comment on rrw, it's probably used for responding separate
> files in distrib=false mode.
> You can start from describing why you need to create own response writer.
>
> On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <lee.a.carroll@googlemail.com
> >
> wrote:
>
> > Ok. My expectation was the response writer would not be used until the
> > final serialization of the result. If my response writer breaks the
> > response writer contract, exactly the way rawResponseWriter does and just
> > out puts a filed value how does that work? Does rawResponseWriter support
> > cloud mode?
> >
> >
> >
> > On 17 April 2018 at 15:55, Mikhail Khludnev <mk...@apache.org> wrote:
> >
> > > That's what should happen.
> > >
> > > Expected mime type application/octet-stream but got application/json.
> > >
> > > Distributed search coordinator expect to merge slave responses in
> javabin
> > > format. But slave's wt indicated json.
> > > As far as I know only javabin might be used to distributed search
> > > underneath. Coordinator itself might yield json.
> > >
> > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
> > lee.a.carroll@googlemail.com
> > > >
> > > wrote:
> > >
> > > > Sure
> > > >
> > > > with 1 shard 1 replica this request works fine
> > > >
> > > > 1. Request URL:
> > > > http://localhost:8983/solr/images/image?q=id:1
> > > > 2. Request Method:
> > > > GET
> > > > 3. Status Code:
> > > > 200 OK
> > > >
> > > > logs are clean
> > > >
> > > > with 2 shards 2 replicas the same request fails and in the logs
> > > >
> > > >
> > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7
> > > > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore;
> > > > [images_shard2_replica_n4] webapp=/solr path=/image
> > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > > http://10.224.30.207:8983/solr/images_shard2_replica_n4/
> > > > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/
> > > > &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=javabin}
> > > > hits=0 status=0 QTime=0
> > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
> > > > x:images_shard1_replica_n1] org.apache.solr.common.SolrException;
> > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
> RemoteSolrException:
> > > > Error
> > > > from server at http://10.224.30.207:8983/
> solr/images_shard2_replica_n4
> > :
> > > > Expected mime type application/octet-stream but got application/json.
> > > > at
> > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
> > > > executeMethod(HttpSolrClient.java:607)
> > > > at
> > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > > > HttpSolrClient.java:255)
> > > > at
> > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > > > HttpSolrClient.java:244)
> > > > at
> > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > > > doRequest(LBHttpSolrClient.java:483)
> > > > at
> > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(
> > > > LBHttpSolrClient.java:413)
> > > > at
> > > > org.apache.solr.handler.component.HttpShardHandlerFactory.
> > > > makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
> > > > at
> > > > org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(
> > > > HttpShardHandler.java:175)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > at java.util.concurrent.Executors$RunnableAdapter.
> > > call(Executors.java:511)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > at
> > > > com.codahale.metrics.InstrumentedExecutorService$
> > > InstrumentedRunnable.run(
> > > > InstrumentedExecutorService.java:176)
> > > > at
> > > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.
> > > > lambda$execute$0(ExecutorUtil.java:188)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > ThreadPoolExecutor.java:1142)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > ThreadPoolExecutor.java:617)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > >
> > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1 r:core_node3
> > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> > > > [images_shard1_replica_n1] webapp=/solr path=/image params={q=id:1}
> > > > status=200 QTime=17
> > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
> > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> > > > [images_shard1_replica_n1] webapp=/solr path=/image
> > > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > > http://10.224.30.207:8983/solr/images_shard1_replica_n1/
> > > > |http://10.224.30.207:7574/solr/images_shard1_replica_n2/
> > > > &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=javabin}
> > > > hits=1 status=0 QTime=2
> > > >
> > > >
> > > > I've implemented getcontenttype simply as
> > > >
> > > > @Override
> > > > public String getContentType(SolrQueryRequest request,
> > > > SolrQueryResponse response) {
> > > >
> > > > return "application/json;charset=utf-8";
> > > > }
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 16 April 2018 at 17:37, Mikhail Khludnev <mk...@apache.org> wrote:
> > > >
> > > > > Lee,
> > > > > It's worth to send a stacktrace for such kind of inquiries.
> > > > > I guess it goes from QueryComponent.mergeIds() or so. Shard
> response
> > > > should
> > > > > contains <uniqueKey> from schema.xml field.
> > > > > I encounter something like this while troubleshooting
> > > > > https://lucene.apache.org/solr/guide/6_6/transforming-
> > > > > result-documents.html#TransformingResultDocuments-
> > > > > CoresandCollectionsinSolrCloud
> > > > >
> > > > >
> > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
> > > > lee.a.carroll@googlemail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > I've created a custom response writer which extends
> > > RawResponseWriter.
> > > > > The
> > > > > > basic operation is to output a single field value from the top
> > > matching
> > > > > doc
> > > > > > as the entire response. This works when shards = 1 but fails when
> > > > shards
> > > > > > are greater than 1.
> > > > > >
> > > > > > I throw an error if the field in question is missing from the top
> > > doc.
> > > > > This
> > > > > > happens when individual shards are being searched and only id and
> > > score
> > > > > are
> > > > > > returned. I'm sure I've committed a basic error.
> > > > > >
> > > > > > Lee C
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sincerely yours
> > > > > Mikhail Khludnev
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > >
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Mikhail Khludnev <mk...@apache.org>.
In distributed search response writer is used twice
https://lucene.apache.org/solr/guide/7_1/distributed-requests.html
once slave node that's where response writer yields "json" content and it
upset aggregator node which is expect only javabin.
I hardly can comment on rrw, it's probably used for responding separate
files in distrib=false mode.
You can start from describing why you need to create own response writer.
On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll <le...@googlemail.com>
wrote:
> Ok. My expectation was the response writer would not be used until the
> final serialization of the result. If my response writer breaks the
> response writer contract, exactly the way rawResponseWriter does and just
> out puts a filed value how does that work? Does rawResponseWriter support
> cloud mode?
>
>
>
> On 17 April 2018 at 15:55, Mikhail Khludnev <mk...@apache.org> wrote:
>
> > That's what should happen.
> >
> > Expected mime type application/octet-stream but got application/json.
> >
> > Distributed search coordinator expect to merge slave responses in javabin
> > format. But slave's wt indicated json.
> > As far as I know only javabin might be used to distributed search
> > underneath. Coordinator itself might yield json.
> >
> > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <
> lee.a.carroll@googlemail.com
> > >
> > wrote:
> >
> > > Sure
> > >
> > > with 1 shard 1 replica this request works fine
> > >
> > > 1. Request URL:
> > > http://localhost:8983/solr/images/image?q=id:1
> > > 2. Request Method:
> > > GET
> > > 3. Status Code:
> > > 200 OK
> > >
> > > logs are clean
> > >
> > > with 2 shards 2 replicas the same request fails and in the logs
> > >
> > >
> > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7
> > > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore;
> > > [images_shard2_replica_n4] webapp=/solr path=/image
> > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > http://10.224.30.207:8983/solr/images_shard2_replica_n4/
> > > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/
> > > &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=javabin}
> > > hits=0 status=0 QTime=0
> > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
> > > x:images_shard1_replica_n1] org.apache.solr.common.SolrException;
> > > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > > Error
> > > from server at http://10.224.30.207:8983/solr/images_shard2_replica_n4
> :
> > > Expected mime type application/octet-stream but got application/json.
> > > at
> > > org.apache.solr.client.solrj.impl.HttpSolrClient.
> > > executeMethod(HttpSolrClient.java:607)
> > > at
> > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > > HttpSolrClient.java:255)
> > > at
> > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > > HttpSolrClient.java:244)
> > > at
> > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > > doRequest(LBHttpSolrClient.java:483)
> > > at
> > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(
> > > LBHttpSolrClient.java:413)
> > > at
> > > org.apache.solr.handler.component.HttpShardHandlerFactory.
> > > makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
> > > at
> > > org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(
> > > HttpShardHandler.java:175)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at java.util.concurrent.Executors$RunnableAdapter.
> > call(Executors.java:511)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at
> > > com.codahale.metrics.InstrumentedExecutorService$
> > InstrumentedRunnable.run(
> > > InstrumentedExecutorService.java:176)
> > > at
> > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.
> > > lambda$execute$0(ExecutorUtil.java:188)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > ThreadPoolExecutor.java:1142)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > ThreadPoolExecutor.java:617)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1 r:core_node3
> > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> > > [images_shard1_replica_n1] webapp=/solr path=/image params={q=id:1}
> > > status=200 QTime=17
> > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
> > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> > > [images_shard1_replica_n1] webapp=/solr path=/image
> > > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > > shards.purpose=4&start=0&fsv=true&shard.url=
> > > http://10.224.30.207:8983/solr/images_shard1_replica_n1/
> > > |http://10.224.30.207:7574/solr/images_shard1_replica_n2/
> > > &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=javabin}
> > > hits=1 status=0 QTime=2
> > >
> > >
> > > I've implemented getcontenttype simply as
> > >
> > > @Override
> > > public String getContentType(SolrQueryRequest request,
> > > SolrQueryResponse response) {
> > >
> > > return "application/json;charset=utf-8";
> > > }
> > >
> > >
> > >
> > >
> > >
> > > On 16 April 2018 at 17:37, Mikhail Khludnev <mk...@apache.org> wrote:
> > >
> > > > Lee,
> > > > It's worth to send a stacktrace for such kind of inquiries.
> > > > I guess it goes from QueryComponent.mergeIds() or so. Shard response
> > > should
> > > > contains <uniqueKey> from schema.xml field.
> > > > I encounter something like this while troubleshooting
> > > > https://lucene.apache.org/solr/guide/6_6/transforming-
> > > > result-documents.html#TransformingResultDocuments-
> > > > CoresandCollectionsinSolrCloud
> > > >
> > > >
> > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
> > > lee.a.carroll@googlemail.com
> > > > >
> > > > wrote:
> > > >
> > > > > I've created a custom response writer which extends
> > RawResponseWriter.
> > > > The
> > > > > basic operation is to output a single field value from the top
> > matching
> > > > doc
> > > > > as the entire response. This works when shards = 1 but fails when
> > > shards
> > > > > are greater than 1.
> > > > >
> > > > > I throw an error if the field in question is missing from the top
> > doc.
> > > > This
> > > > > happens when individual shards are being searched and only id and
> > score
> > > > are
> > > > > returned. I'm sure I've committed a basic error.
> > > > >
> > > > > Lee C
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sincerely yours
> > > > Mikhail Khludnev
> > > >
> > >
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> >
>
--
Sincerely yours
Mikhail Khludnev
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Lee Carroll <le...@googlemail.com>.
Ok. My expectation was the response writer would not be used until the
final serialization of the result. If my response writer breaks the
response writer contract, exactly the way rawResponseWriter does and just
out puts a filed value how does that work? Does rawResponseWriter support
cloud mode?
On 17 April 2018 at 15:55, Mikhail Khludnev <mk...@apache.org> wrote:
> That's what should happen.
>
> Expected mime type application/octet-stream but got application/json.
>
> Distributed search coordinator expect to merge slave responses in javabin
> format. But slave's wt indicated json.
> As far as I know only javabin might be used to distributed search
> underneath. Coordinator itself might yield json.
>
> On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <lee.a.carroll@googlemail.com
> >
> wrote:
>
> > Sure
> >
> > with 1 shard 1 replica this request works fine
> >
> > 1. Request URL:
> > http://localhost:8983/solr/images/image?q=id:1
> > 2. Request Method:
> > GET
> > 3. Status Code:
> > 200 OK
> >
> > logs are clean
> >
> > with 2 shards 2 replicas the same request fails and in the logs
> >
> >
> > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7
> > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore;
> > [images_shard2_replica_n4] webapp=/solr path=/image
> > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > shards.purpose=4&start=0&fsv=true&shard.url=
> > http://10.224.30.207:8983/solr/images_shard2_replica_n4/
> > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/
> > &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=javabin}
> > hits=0 status=0 QTime=0
> > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
> > x:images_shard1_replica_n1] org.apache.solr.common.SolrException;
> > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > Error
> > from server at http://10.224.30.207:8983/solr/images_shard2_replica_n4:
> > Expected mime type application/octet-stream but got application/json.
> > at
> > org.apache.solr.client.solrj.impl.HttpSolrClient.
> > executeMethod(HttpSolrClient.java:607)
> > at
> > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > HttpSolrClient.java:255)
> > at
> > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > HttpSolrClient.java:244)
> > at
> > org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > doRequest(LBHttpSolrClient.java:483)
> > at
> > org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(
> > LBHttpSolrClient.java:413)
> > at
> > org.apache.solr.handler.component.HttpShardHandlerFactory.
> > makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
> > at
> > org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(
> > HttpShardHandler.java:175)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > com.codahale.metrics.InstrumentedExecutorService$
> InstrumentedRunnable.run(
> > InstrumentedExecutorService.java:176)
> > at
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.
> > lambda$execute$0(ExecutorUtil.java:188)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1 r:core_node3
> > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> > [images_shard1_replica_n1] webapp=/solr path=/image params={q=id:1}
> > status=200 QTime=17
> > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
> > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> > [images_shard1_replica_n1] webapp=/solr path=/image
> > params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> > shards.purpose=4&start=0&fsv=true&shard.url=
> > http://10.224.30.207:8983/solr/images_shard1_replica_n1/
> > |http://10.224.30.207:7574/solr/images_shard1_replica_n2/
> > &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=javabin}
> > hits=1 status=0 QTime=2
> >
> >
> > I've implemented getcontenttype simply as
> >
> > @Override
> > public String getContentType(SolrQueryRequest request,
> > SolrQueryResponse response) {
> >
> > return "application/json;charset=utf-8";
> > }
> >
> >
> >
> >
> >
> > On 16 April 2018 at 17:37, Mikhail Khludnev <mk...@apache.org> wrote:
> >
> > > Lee,
> > > It's worth to send a stacktrace for such kind of inquiries.
> > > I guess it goes from QueryComponent.mergeIds() or so. Shard response
> > should
> > > contains <uniqueKey> from schema.xml field.
> > > I encounter something like this while troubleshooting
> > > https://lucene.apache.org/solr/guide/6_6/transforming-
> > > result-documents.html#TransformingResultDocuments-
> > > CoresandCollectionsinSolrCloud
> > >
> > >
> > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
> > lee.a.carroll@googlemail.com
> > > >
> > > wrote:
> > >
> > > > I've created a custom response writer which extends
> RawResponseWriter.
> > > The
> > > > basic operation is to output a single field value from the top
> matching
> > > doc
> > > > as the entire response. This works when shards = 1 but fails when
> > shards
> > > > are greater than 1.
> > > >
> > > > I throw an error if the field in question is missing from the top
> doc.
> > > This
> > > > happens when individual shards are being searched and only id and
> score
> > > are
> > > > returned. I'm sure I've committed a basic error.
> > > >
> > > > Lee C
> > > >
> > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > >
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Mikhail Khludnev <mk...@apache.org>.
That's what should happen.
Expected mime type application/octet-stream but got application/json.
Distributed search coordinator expect to merge slave responses in javabin
format. But slave's wt indicated json.
As far as I know only javabin might be used to distributed search
underneath. Coordinator itself might yield json.
On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll <le...@googlemail.com>
wrote:
> Sure
>
> with 1 shard 1 replica this request works fine
>
> 1. Request URL:
> http://localhost:8983/solr/images/image?q=id:1
> 2. Request Method:
> GET
> 3. Status Code:
> 200 OK
>
> logs are clean
>
> with 2 shards 2 replicas the same request fails and in the logs
>
>
> INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7
> x:images_shard2_replica_n4] org.apache.solr.core.SolrCore;
> [images_shard2_replica_n4] webapp=/solr path=/image
> params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> shards.purpose=4&start=0&fsv=true&shard.url=
> http://10.224.30.207:8983/solr/images_shard2_replica_n4/
> |http://10.224.30.207:7574/solr/images_shard2_replica_n6/
> &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=javabin}
> hits=0 status=0 QTime=0
> ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
> x:images_shard1_replica_n1] org.apache.solr.common.SolrException;
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error
> from server at http://10.224.30.207:8983/solr/images_shard2_replica_n4:
> Expected mime type application/octet-stream but got application/json.
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.
> executeMethod(HttpSolrClient.java:607)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> HttpSolrClient.java:255)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> HttpSolrClient.java:244)
> at
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> doRequest(LBHttpSolrClient.java:483)
> at
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(
> LBHttpSolrClient.java:413)
> at
> org.apache.solr.handler.component.HttpShardHandlerFactory.
> makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
> at
> org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(
> HttpShardHandler.java:175)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(
> InstrumentedExecutorService.java:176)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.
> lambda$execute$0(ExecutorUtil.java:188)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> INFO - 2018-04-17 13:20:32.056; [c:images s:shard1 r:core_node3
> x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> [images_shard1_replica_n1] webapp=/solr path=/image params={q=id:1}
> status=200 QTime=17
> INFO - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
> x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
> [images_shard1_replica_n1] webapp=/solr path=/image
> params={df=text&distrib=false&qt=/image&fl=id&fl=score&
> shards.purpose=4&start=0&fsv=true&shard.url=
> http://10.224.30.207:8983/solr/images_shard1_replica_n1/
> |http://10.224.30.207:7574/solr/images_shard1_replica_n2/
> &rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=javabin}
> hits=1 status=0 QTime=2
>
>
> I've implemented getcontenttype simply as
>
> @Override
> public String getContentType(SolrQueryRequest request,
> SolrQueryResponse response) {
>
> return "application/json;charset=utf-8";
> }
>
>
>
>
>
> On 16 April 2018 at 17:37, Mikhail Khludnev <mk...@apache.org> wrote:
>
> > Lee,
> > It's worth to send a stacktrace for such kind of inquiries.
> > I guess it goes from QueryComponent.mergeIds() or so. Shard response
> should
> > contains <uniqueKey> from schema.xml field.
> > I encounter something like this while troubleshooting
> > https://lucene.apache.org/solr/guide/6_6/transforming-
> > result-documents.html#TransformingResultDocuments-
> > CoresandCollectionsinSolrCloud
> >
> >
> > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <
> lee.a.carroll@googlemail.com
> > >
> > wrote:
> >
> > > I've created a custom response writer which extends RawResponseWriter.
> > The
> > > basic operation is to output a single field value from the top matching
> > doc
> > > as the entire response. This works when shards = 1 but fails when
> shards
> > > are greater than 1.
> > >
> > > I throw an error if the field in question is missing from the top doc.
> > This
> > > happens when individual shards are being searched and only id and score
> > are
> > > returned. I'm sure I've committed a basic error.
> > >
> > > Lee C
> > >
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> >
>
--
Sincerely yours
Mikhail Khludnev
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Lee Carroll <le...@googlemail.com>.
Sure
with 1 shard 1 replica this request works fine
1. Request URL:
http://localhost:8983/solr/images/image?q=id:1
2. Request Method:
GET
3. Status Code:
200 OK
logs are clean
with 2 shards 2 replicas the same request fails and in the logs
INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7
x:images_shard2_replica_n4] org.apache.solr.core.SolrCore;
[images_shard2_replica_n4] webapp=/solr path=/image
params={df=text&distrib=false&qt=/image&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=
http://10.224.30.207:8983/solr/images_shard2_replica_n4/|http://10.224.30.207:7574/solr/images_shard2_replica_n6/&rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=javabin}
hits=0 status=0 QTime=0
ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
x:images_shard1_replica_n1] org.apache.solr.common.SolrException;
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://10.224.30.207:8983/solr/images_shard2_replica_n4:
Expected mime type application/octet-stream but got application/json.
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at
org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
at
org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:175)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO - 2018-04-17 13:20:32.056; [c:images s:shard1 r:core_node3
x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
[images_shard1_replica_n1] webapp=/solr path=/image params={q=id:1}
status=200 QTime=17
INFO - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3
x:images_shard1_replica_n1] org.apache.solr.core.SolrCore;
[images_shard1_replica_n1] webapp=/solr path=/image
params={df=text&distrib=false&qt=/image&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=
http://10.224.30.207:8983/solr/images_shard1_replica_n1/|http://10.224.30.207:7574/solr/images_shard1_replica_n2/&rows=10&version=2&q=id:1&NOW=1523971232039&isShard=true&wt=javabin}
hits=1 status=0 QTime=2
I've implemented getcontenttype simply as
@Override
public String getContentType(SolrQueryRequest request,
SolrQueryResponse response) {
return "application/json;charset=utf-8";
}
On 16 April 2018 at 17:37, Mikhail Khludnev <mk...@apache.org> wrote:
> Lee,
> It's worth to send a stacktrace for such kind of inquiries.
> I guess it goes from QueryComponent.mergeIds() or so. Shard response should
> contains <uniqueKey> from schema.xml field.
> I encounter something like this while troubleshooting
> https://lucene.apache.org/solr/guide/6_6/transforming-
> result-documents.html#TransformingResultDocuments-
> CoresandCollectionsinSolrCloud
>
>
> On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <lee.a.carroll@googlemail.com
> >
> wrote:
>
> > I've created a custom response writer which extends RawResponseWriter.
> The
> > basic operation is to output a single field value from the top matching
> doc
> > as the entire response. This works when shards = 1 but fails when shards
> > are greater than 1.
> >
> > I throw an error if the field in question is missing from the top doc.
> This
> > happens when individual shards are being searched and only id and score
> are
> > returned. I'm sure I've committed a basic error.
> >
> > Lee C
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
Re: custom response writer which extends RawResponseWriter fails when
shards > 1
Posted by Mikhail Khludnev <mk...@apache.org>.
Lee,
It's worth to send a stacktrace for such kind of inquiries.
I guess it goes from QueryComponent.mergeIds() or so. Shard response should
contains <uniqueKey> from schema.xml field.
I encounter something like this while troubleshooting
https://lucene.apache.org/solr/guide/6_6/transforming-result-documents.html#TransformingResultDocuments-CoresandCollectionsinSolrCloud
On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll <le...@googlemail.com>
wrote:
> I've created a custom response writer which extends RawResponseWriter. The
> basic operation is to output a single field value from the top matching doc
> as the entire response. This works when shards = 1 but fails when shards
> are greater than 1.
>
> I throw an error if the field in question is missing from the top doc. This
> happens when individual shards are being searched and only id and score are
> returned. I'm sure I've committed a basic error.
>
> Lee C
>
--
Sincerely yours
Mikhail Khludnev