You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by iboppana <in...@cmgdigital.com> on 2010/06/15 18:59:43 UTC
DatImportHandler and cron issue
Hi All,
We are trying implement solr for our newspapers site search.
To build out the index with all the articles published so far, we are
running script which send the request to dataimport handler with different
dates.
What we are seeing is the request is dispatched to solr server,but its not
being processed.
Just wanted to check if its some kind of threading issues, and whats the
best approach to achieve this.
We are sleeping for 75 secs between the requests,
while (($date+=86400) < $now) {
$curdate=strftime("%D", localtime($date));
print "Updating index for $curdate\n";
$curdate=uri_escape($curdate);
my $url =
'http://test.solr.ddtc.cmgdigital.com:8080/solr/npmetrosearch_statesman/dataimport?command=full-import&entity=initialLoad&clean=false&commit=true&forDate='
. $curdate .
'&numArticles=-1&server=app5&site=statesman&articleTypes=story,slideshow,video,poll,specialArticle,list';
print "Sending: $url\n";
#if (system("wget -q -O - \'$url\' | egrep -q \'$regex_pat\'")) {
if (system("curl -s \'$url\' | egrep -q \'$regex_pat\'")) {
print "Failed to match expected regex reply: \"$regex_pat\"\n";
exit 1;
}
sleep 75;
}
This is what we are seeing on the server logs
2010-06-14 12:51:01,328 INFO [org.apache.solr.core.SolrCore]
(http-0.0.0.0-8080-1) [npmetrosearch_statesman] webapp=/solr
path=/dataimport
params={site=statesman&forDate=03/24/10&articleTypes=story,slideshow,video,poll,specialArticle,list&clean=false&commit=true&entity=initialLoad&command=full-import&numArticles=-1&server=app5}
status=0 QTime=0
2010-06-14 12:51:01,329 INFO
[org.apache.solr.handler.dataimport.DataImporter] (Thread-378) Starting Full
Import
2010-06-14 12:51:01,332 INFO
[org.apache.solr.handler.dataimport.SolrWriter] (Thread-378) Read
dataimport.properties
2010-06-14 12:51:01,425 INFO
[org.apache.solr.handler.dataimport.DocBuilder] (Thread-378) Time taken =
0:0:0.93
2010-06-14 12:51:16,338 INFO [org.apache.solr.core.SolrCore]
(http-0.0.0.0-8080-1) [npmetrosearch_statesman] webapp=/solr
path=/dataimport
params={site=statesman&forDate=03/25/10&articleTypes=story,slideshow,video,poll,specialArticle,list&clean=false&commit=true&entity=initialLoad&command=full-import&numArticles=-1&server=app5}
status=0 QTime=0
2010-06-14 12:51:16,338 INFO
[org.apache.solr.handler.dataimport.DataImporter] (Thread-379) Starting Full
Import
2010-06-14 12:51:16,338 INFO
[org.apache.solr.handler.dataimport.SolrWriter] (Thread-379) Read
dataimport.properties
2010-06-14 12:51:16,465 INFO
[org.apache.solr.handler.dataimport.DocBuilder] (Thread-379) Time taken =
0:0:0.126
Appreciate any thoughts on this.
Thanks
Indrani
--
View this message in context: http://lucene.472066.n3.nabble.com/DatImportHandler-and-cron-issue-tp897698p897698.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DatImportHandler and cron issue
Posted by Govind Kanshi <go...@gmail.com>.
How did you verify it was not processed? Did you
1. Query for docs - with no results
2. Use Solr Admin tool?
3. Bypass data import handler and see if the doc post/commit works.
On Tue, Jun 15, 2010 at 10:29 PM, iboppana
<in...@cmgdigital.com>wrote:
>
> Hi All,
>
> We are trying implement solr for our newspapers site search.
> To build out the index with all the articles published so far, we are
> running script which send the request to dataimport handler with different
> dates.
> What we are seeing is the request is dispatched to solr server,but its not
> being processed.
> Just wanted to check if its some kind of threading issues, and whats the
> best approach to achieve this.
>
> We are sleeping for 75 secs between the requests,
>
>
> while (($date+=86400) < $now) {
> $curdate=strftime("%D", localtime($date));
>
> print "Updating index for $curdate\n";
>
> $curdate=uri_escape($curdate);
>
> my $url =
> '
> http://test.solr.ddtc.cmgdigital.com:8080/solr/npmetrosearch_statesman/dataimport?command=full-import&entity=initialLoad&clean=false&commit=true&forDate=
> '
> . $curdate .
>
> '&numArticles=-1&server=app5&site=statesman&articleTypes=story,slideshow,video,poll,specialArticle,list';
>
> print "Sending: $url\n";
>
>
> #if (system("wget -q -O - \'$url\' | egrep -q \'$regex_pat\'")) {
> if (system("curl -s \'$url\' | egrep -q \'$regex_pat\'")) {
> print "Failed to match expected regex reply: \"$regex_pat\"\n";
> exit 1;
> }
>
> sleep 75;
> }
>
>
>
>
> This is what we are seeing on the server logs
> 2010-06-14 12:51:01,328 INFO [org.apache.solr.core.SolrCore]
> (http-0.0.0.0-8080-1) [npmetrosearch_statesman] webapp=/solr
> path=/dataimport
>
> params={site=statesman&forDate=03/24/10&articleTypes=story,slideshow,video,poll,specialArticle,list&clean=false&commit=true&entity=initialLoad&command=full-import&numArticles=-1&server=app5}
> status=0 QTime=0
> 2010-06-14 12:51:01,329 INFO
> [org.apache.solr.handler.dataimport.DataImporter] (Thread-378) Starting
> Full
> Import
> 2010-06-14 12:51:01,332 INFO
> [org.apache.solr.handler.dataimport.SolrWriter] (Thread-378) Read
> dataimport.properties
> 2010-06-14 12:51:01,425 INFO
> [org.apache.solr.handler.dataimport.DocBuilder] (Thread-378) Time taken =
> 0:0:0.93
> 2010-06-14 12:51:16,338 INFO [org.apache.solr.core.SolrCore]
> (http-0.0.0.0-8080-1) [npmetrosearch_statesman] webapp=/solr
> path=/dataimport
>
> params={site=statesman&forDate=03/25/10&articleTypes=story,slideshow,video,poll,specialArticle,list&clean=false&commit=true&entity=initialLoad&command=full-import&numArticles=-1&server=app5}
> status=0 QTime=0
> 2010-06-14 12:51:16,338 INFO
> [org.apache.solr.handler.dataimport.DataImporter] (Thread-379) Starting
> Full
> Import
> 2010-06-14 12:51:16,338 INFO
> [org.apache.solr.handler.dataimport.SolrWriter] (Thread-379) Read
> dataimport.properties
> 2010-06-14 12:51:16,465 INFO
> [org.apache.solr.handler.dataimport.DocBuilder] (Thread-379) Time taken =
> 0:0:0.126
>
> Appreciate any thoughts on this.
>
> Thanks
> Indrani
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/DatImportHandler-and-cron-issue-tp897698p897698.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: DatImportHandler and cron issue
Posted by Chris Hostetter <ho...@fucit.org>.
: What we are seeing is the request is dispatched to solr server,but its not
: being processed.
you'll have to explain what you mean by "not being processed" ?
According to your logs, DIH is in fact working and logging it's
progress...
: 2010-06-14 12:51:01,328 INFO [org.apache.solr.core.SolrCore]
: (http-0.0.0.0-8080-1) [npmetrosearch_statesman] webapp=/solr
: path=/dataimport
: params={site=statesman&forDate=03/24/10&articleTypes=story,slideshow,video,poll,specialArticle,list&clean=false&commit=true&entity=initialLoad&command=full-import&numArticles=-1&server=app5}
: status=0 QTime=0
: 2010-06-14 12:51:01,329 INFO
: [org.apache.solr.handler.dataimport.DataImporter] (Thread-378) Starting Full
: Import
: 2010-06-14 12:51:01,332 INFO
: [org.apache.solr.handler.dataimport.SolrWriter] (Thread-378) Read
: dataimport.properties
: 2010-06-14 12:51:01,425 INFO
: [org.apache.solr.handler.dataimport.DocBuilder] (Thread-378) Time taken =
: 0:0:0.93
-Hoss