RE: [Marketing Mail] Re: Request return behavior when trigger DataImportHandler updates

[Nathan Friend <na...@Maximuscanada.ca>]

Thanks for the reply, Shawn.  One additional detail I should have mentioned - in both scenarios, I'm making a POST request, not a GET.  Even when using a POST, Solr seems to be able to accept URL query parameters.

I'm making test requests using Postman (https://www.getpostman.com/), so it's also possible this issue is related to how Postman is sending the request (although I doubt it).

Either way, it sounds like relying on the synchronous behavior of the HTTP request is not a good option.  I've been looking into the "onImportEnd" event listener functionality and think this feature will provide a more robust way of generating notifications when imports have finished.

Thanks!

Nathan

-----Original Message-----
From: Shawn Heisey [mailto:apache@elyograg.org] 
Sent: December 1, 2017 3:04 PM
To: solr-user@lucene.apache.org
Subject: [Marketing Mail] Re: Request return behavior when trigger DataImportHandler updates

On 12/1/2017 9:37 AM, Nathan Friend wrote:
> When triggering the DataImportHandler to update via an HTTP request, I've noticed the handler behaves differently depending on how I specify the request parameters.
>
> If I use query parameters, e.g. http://localhost:8983/solr/mycore/dataimport?command=full-import, the HTTP request is not fulfilled until the import is completed.
>
> However, if I instead make a request to http://localhost:8983/solr/mycore/dataimport with a form-data key of "command"="full-import", the HTTP request returns immediately. In this case, the documentation seems to suggest the best way to detect when the import has completed is to poll Solr for status updates until the status switches from "busy" back to "idle".

At one time, way back in version 1.4 and 3.2, I was making such requests from a Perl program, and they were definitely GET requests.  These requests did NOT wait until the import completed.

I just checked my most recent program, which uses SolrJ, and I discovered that I am using Method.POST there, so I can't confirm or deny what you're seeing yet.  At one point, I went through all my SolrJ code and made sure all requests were using POST, so that I would never run into the limitations on the URL size.  It wasn't really necessary for my code that starts DIH, but I was making the change everywhere.

> Is this difference in behavior intentional?  I would like to be able to rely on the first behavior - to expect that the import will be finished when the HTTP call returns.  However, I can't seem to find any documentation on this difference; I'd prefer not to have my component rely on what might be a convenient bug :).

As far as I know, Solr should be unaware of any difference between these ways of supplying the parameters, because the servlet container (usually
Jetty) should just make parameters available to Solr, no matter how it received them.  I think that the dataimport handler has always been async in nature.  I've NEVER seen an import command wait to respond until the import is complete.  My imports take many hours to run.  If that's what you're seeing happen with a GET request, I'd call that a bug.  We need to confirm the problem in the most recent versions before filing an issue.

Thanks,
Shawn


Confidentiality Notice:
This electronic transmission, and any documents attached to it, may contain confidential information belonging to the sender. This information is intended solely for the use of the individual or entity named above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or the taking of any action in reliance upon the contents of this information is prohibited. If you have received this transmission in error, please notify the sender immediately and delete the message and all documents.