You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Lasitha Wattaladeniya <wa...@gmail.com> on 2017/02/13 04:05:27 UTC
ConcurrentUpdateSolrServer update response doesn't reflect correct
response status
Hi devs,
I'm using the ConcurrentUpdateSolrServer to add some documents to solr.
While I'm doing so I see some 'bad request" errors printed in the console
due to incorrect solrinputdocuments.
The issue is I can't capture this issue by reading updateresonse status.
I saw some one had stated that this is due to ConcurrentUpdateSolrServer
uses a queue to buffer the documents and then later call update method.
Isn't there any way to capture the correct response?
Regards,
Lasitha
Re: ConcurrentUpdateSolrServer update response doesn't reflect
correct response status
Posted by Lasitha Wattaladeniya <wa...@gmail.com>.
Hi Shawn,
Thanks for the detailed explanation, really informative. However after
further analyzing I found that I can use HttpSolrServer instead of
ConcurrentUpdateSolrServer,
I'm handling the concurrency by my self. But the issue with
ConcurrentUpdateSolrServer
is still there, seem like an interesting topic to look into.
Regards,
Lasitha
Lasitha Wattaladeniya
Software Engineer
Mobile : +6593896893
Blog : techreadme.blogspot.com
On Mon, Feb 13, 2017 at 10:01 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 2/12/2017 10:51 PM, Lasitha Wattaladeniya wrote:
> > Thanks for the reply. But even if I add a single document, the response
> > doesn't reflect the correct response status. From the given solr console
> > ui, it shows status 400 for the bad requests (when I try to index
> document
> > with no fields <doc></doc>). But when I do the same using solrj,
> response
> > is 0 eventhough the console prints it as a bad request with status 400.
>
> The concurrent client swallows all indexing exceptions. This is how it
> is designed. In fact, it's potentially even worse than you might expect
> -- the client not only always returns a success for updates sent, it
> returns immediately, often before any updates are sent to the server.
> The updates are sent in the background. If another request type is
> sent, like a commit, then the client will wait for all queued updates to
> be sent before it sends that request and returns. If an error is
> encountered during an explicit commit, I believe that the client *is*
> notified about that, but I have not tested this.
>
> If you want multi-threaded indexing *and* error detection, you're going
> to need to handle the multiple threads in your own program, using the
> Http or Cloud client object instead. The Concurrent object is good for
> initial bulk indexing, but not for the kind of indexing where you want
> to be informed about errors.
>
> I'm told that if you override the "handleError" method, you might be
> able to make it work how you want, but I was unable to figure out how to
> modify that method to achieve that result.
>
> I filed an issue and even came up with an idea for a fix, but nobody was
> excited about it. Because of the design of the client, it wasn't
> possible to guarantee that no further indexing had occurred after the
> first error encountered.
>
> https://issues.apache.org/jira/browse/SOLR-3284
>
> Thanks,
> Shawn
>
>
Re: ConcurrentUpdateSolrServer update response doesn't reflect
correct response status
Posted by Shawn Heisey <ap...@elyograg.org>.
On 2/12/2017 10:51 PM, Lasitha Wattaladeniya wrote:
> Thanks for the reply. But even if I add a single document, the response
> doesn't reflect the correct response status. From the given solr console
> ui, it shows status 400 for the bad requests (when I try to index document
> with no fields <doc></doc>). But when I do the same using solrj, response
> is 0 eventhough the console prints it as a bad request with status 400.
The concurrent client swallows all indexing exceptions. This is how it
is designed. In fact, it's potentially even worse than you might expect
-- the client not only always returns a success for updates sent, it
returns immediately, often before any updates are sent to the server.
The updates are sent in the background. If another request type is
sent, like a commit, then the client will wait for all queued updates to
be sent before it sends that request and returns. If an error is
encountered during an explicit commit, I believe that the client *is*
notified about that, but I have not tested this.
If you want multi-threaded indexing *and* error detection, you're going
to need to handle the multiple threads in your own program, using the
Http or Cloud client object instead. The Concurrent object is good for
initial bulk indexing, but not for the kind of indexing where you want
to be informed about errors.
I'm told that if you override the "handleError" method, you might be
able to make it work how you want, but I was unable to figure out how to
modify that method to achieve that result.
I filed an issue and even came up with an idea for a fix, but nobody was
excited about it. Because of the design of the client, it wasn't
possible to guarantee that no further indexing had occurred after the
first error encountered.
https://issues.apache.org/jira/browse/SOLR-3284
Thanks,
Shawn
Re: ConcurrentUpdateSolrServer update response doesn't reflect
correct response status
Posted by Lasitha Wattaladeniya <wa...@gmail.com>.
Hi Eric,
Thanks for the reply. But even if I add a single document, the response
doesn't reflect the correct response status. From the given solr console
ui, it shows status 400 for the bad requests (when I try to index document
with no fields <doc></doc>). But when I do the same using solrj, response
is 0 eventhough the console prints it as a bad request with status 400.
P.S: I'm using solrj 4.10.4
Regards,
Lasitha
On 13 Feb 2017 13:33, "Erick Erickson" <er...@gmail.com> wrote:
Closest that I know of is:
https://issues.apache.org/jira/browse/SOLR-445, available since Solr
6.1
Best,
Erick
On Sun, Feb 12, 2017 at 8:05 PM, Lasitha Wattaladeniya
<wa...@gmail.com> wrote:
> Hi devs,
>
> I'm using the ConcurrentUpdateSolrServer to add some documents to solr.
> While I'm doing so I see some 'bad request" errors printed in the console
> due to incorrect solrinputdocuments.
>
> The issue is I can't capture this issue by reading updateresonse status.
> I saw some one had stated that this is due to ConcurrentUpdateSolrServer
> uses a queue to buffer the documents and then later call update method.
>
> Isn't there any way to capture the correct response?
>
> Regards,
> Lasitha
Re: ConcurrentUpdateSolrServer update response doesn't reflect
correct response status
Posted by Erick Erickson <er...@gmail.com>.
Closest that I know of is:
https://issues.apache.org/jira/browse/SOLR-445, available since Solr
6.1
Best,
Erick
On Sun, Feb 12, 2017 at 8:05 PM, Lasitha Wattaladeniya
<wa...@gmail.com> wrote:
> Hi devs,
>
> I'm using the ConcurrentUpdateSolrServer to add some documents to solr.
> While I'm doing so I see some 'bad request" errors printed in the console
> due to incorrect solrinputdocuments.
>
> The issue is I can't capture this issue by reading updateresonse status.
> I saw some one had stated that this is due to ConcurrentUpdateSolrServer
> uses a queue to buffer the documents and then later call update method.
>
> Isn't there any way to capture the correct response?
>
> Regards,
> Lasitha