You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Koji Sekiguchi <ko...@r.email.ne.jp> on 2010/08/31 18:11:41 UTC
missing part folder - how to debug?
Hello,
We are using Hadoop to make Solr index. We are using SOLR-1301
that was first contributed by Andrzej:
https://issues.apache.org/jira/browse/SOLR-1301
It works great on testing environment, 4 servers.
Today, we run it on production environment, 320 servers.
We run 5120 reducers (16 per server). This results 5120 indexes
i.e. part-XXXXX folders should be created. But about 20 part
folders were missing, and Hadoop didn't produce any error logs.
How can we investigate/debug this problem?
Any pointers, experiences would be highly appreciated!
Thanks,
Koji
--
http://www.rondhuit.com/en/
Re: missing part folder - how to debug?
Posted by Alex Baranau <al...@gmail.com>.
Hi,
Adding Solr user list.
We used similar approach to the one in this patch but with Hadoop Streaming.
Did you determine that indices are really missing? I mean did you find
missing documents in the output indices?
Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - HBase
2010/8/31 Koji Sekiguchi <ko...@r.email.ne.jp>
> Hello,
>
> We are using Hadoop to make Solr index. We are using SOLR-1301
> that was first contributed by Andrzej:
>
> https://issues.apache.org/jira/browse/SOLR-1301
>
> It works great on testing environment, 4 servers.
> Today, we run it on production environment, 320 servers.
> We run 5120 reducers (16 per server). This results 5120 indexes
> i.e. part-XXXXX folders should be created. But about 20 part
> folders were missing, and Hadoop didn't produce any error logs.
> How can we investigate/debug this problem?
>
> Any pointers, experiences would be highly appreciated!
>
> Thanks,
>
> Koji
>
> --
> http://www.rondhuit.com/en/
>
>
Re: missing part folder - how to debug?
Posted by Alex Baranau <al...@gmail.com>.
Hi,
Adding Solr user list.
We used similar approach to the one in this patch but with Hadoop Streaming.
Did you determine that indices are really missing? I mean did you find
missing documents in the output indices?
Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - HBase
2010/8/31 Koji Sekiguchi <ko...@r.email.ne.jp>
> Hello,
>
> We are using Hadoop to make Solr index. We are using SOLR-1301
> that was first contributed by Andrzej:
>
> https://issues.apache.org/jira/browse/SOLR-1301
>
> It works great on testing environment, 4 servers.
> Today, we run it on production environment, 320 servers.
> We run 5120 reducers (16 per server). This results 5120 indexes
> i.e. part-XXXXX folders should be created. But about 20 part
> folders were missing, and Hadoop didn't produce any error logs.
> How can we investigate/debug this problem?
>
> Any pointers, experiences would be highly appreciated!
>
> Thanks,
>
> Koji
>
> --
> http://www.rondhuit.com/en/
>
>