You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Carl Roberts <ca...@gmail.com> on 2015/01/28 01:47:41 UTC
Running multiple full-import commands via curl in a script
Hi,
I am attempting to run all these curl commands from a script so that I
can put them in a crontab job, however, it seems that only the first one
executes and the other ones return with an error (below):
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-import&clean=false&entity=cve-2002"
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-import&clean=false&entity=cve-2003"
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-import&clean=false&entity=cve-2004"
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-import&clean=false&entity=cve-2005"
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-import&clean=false&entity=cve-2006"
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-import&clean=false&entity=cve-2007"
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-import&clean=false&entity=cve-2008"
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-import&clean=false&entity=cve-2009"
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-import&clean=false&entity=cve-2010"
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-import&clean=false&entity=cve-2011"
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-import&clean=false&entity=cve-2012"
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-import&clean=false&entity=cve-2013"
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-import&clean=false&entity=cve-2014"
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-import&clean=false&entity=cve-2015"
curl
"http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=delta-import&clean=false&entity=cve-last"
error:
*A command is still running...*
Question: Is there a way to queue the other requests in Solr so that
they run as soon as the previous one is done? If not, how would you
recommend I do this?
Many thanks in advance,
Joe
Re: Running multiple full-import commands via curl in a script
Posted by Carl Roberts <ca...@gmail.com>.
Thanks Mikhail - synchronous=true works like a charm...:)
On 1/28/15, 5:16 AM, Mikhail Khludnev wrote:
> Literally, queue can be done by submitting as is (async) and polling
> command status. However, giving
> https://github.com/apache/lucene-solr/blob/trunk/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DataImportHandler.java#L200
> you can try to add &synchronous=true&... that should hang request until
> it's completed.
> The other question is how run requests in parallel which is explicitly
> violated by
> https://github.com/apache/lucene-solr/blob/trunk/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DataImportHandler.java#L173
> The only workaround I can suggest is to duplicate DIH definitions in solr
> config
> <requestHandler name="/dataimport" class="solr.DataImportHandler"> ...
> <requestHandler name="/dataimport2" class="solr.DataImportHandler"> ...
> <requestHandler name="/dataimport3" class="solr.DataImportHandler"> ...
> ...
> then those guys should be able to handle own request in parallel. Nasty
> stuff..
> have a good hack
>
> On Wed, Jan 28, 2015 at 3:47 AM, Carl Roberts <carl.roberts.zapata@gmail.com
>> wrote:
>> Hi,
>>
>> I am attempting to run all these curl commands from a script so that I can
>> put them in a crontab job, however, it seems that only the first one
>> executes and the other ones return with an error (below):
>>
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2002"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2003"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2004"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2005"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2006"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2007"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2008"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2009"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2010"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2011"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2012"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2013"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2014"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2015"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> delta-import&clean=false&entity=cve-last"
>>
>> error:
>>
>> *A command is still running...*
>>
>> Question: Is there a way to queue the other requests in Solr so that they
>> run as soon as the previous one is done? If not, how would you recommend I
>> do this?
>>
>> Many thanks in advance,
>>
>> Joe
>>
>>
>>
>
Re: Running multiple full-import commands via curl in a script
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Literally, queue can be done by submitting as is (async) and polling
command status. However, giving
https://github.com/apache/lucene-solr/blob/trunk/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DataImportHandler.java#L200
you can try to add &synchronous=true&... that should hang request until
it's completed.
The other question is how run requests in parallel which is explicitly
violated by
https://github.com/apache/lucene-solr/blob/trunk/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DataImportHandler.java#L173
The only workaround I can suggest is to duplicate DIH definitions in solr
config
<requestHandler name="/dataimport" class="solr.DataImportHandler"> ...
<requestHandler name="/dataimport2" class="solr.DataImportHandler"> ...
<requestHandler name="/dataimport3" class="solr.DataImportHandler"> ...
...
then those guys should be able to handle own request in parallel. Nasty
stuff..
have a good hack
On Wed, Jan 28, 2015 at 3:47 AM, Carl Roberts <carl.roberts.zapata@gmail.com
> wrote:
> Hi,
>
> I am attempting to run all these curl commands from a script so that I can
> put them in a crontab job, however, it seems that only the first one
> executes and the other ones return with an error (below):
>
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2002"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2003"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2004"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2005"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2006"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2007"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2008"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2009"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2010"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2011"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2012"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2013"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2014"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2015"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> delta-import&clean=false&entity=cve-last"
>
> error:
>
> *A command is still running...*
>
> Question: Is there a way to queue the other requests in Solr so that they
> run as soon as the previous one is done? If not, how would you recommend I
> do this?
>
> Many thanks in advance,
>
> Joe
>
>
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>