You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Hiroshi Tatsumi <ho...@comet.ocn.ne.jp> on 2014/04/09 05:35:52 UTC
Handle SolrServerException in SolrConnecotor, HttpPoster
Hi,
I'm using MCF1.5.1 and Solr4.6.1.
When I use SolrConnecotor, sometimes SolrServerException occurs.
Normally, SolrServerException is caught by HttpPoster, line 950.
But in my case, the inner exception of SolrServerException is
SocketTimeoutException, not SocketException.
So, activityCode is set to failed, then mcf interrupt the crawl process.
In this case, I expect that mcf shouldn't interrupt the crawl process.
Mcf should skip the invalid file.
Could you modify the mcf code, or provide a option?
If you could, I'm glad to being modified in the future version.
Regards,
Hiroshi Tatsumi
Re: Handle SolrServerException in SolrConnecotor, HttpPoster
Posted by Karl Wright <da...@gmail.com>.
Hmm.
I looked at the code in order to make this change, and am now puzzled.
Please see my comments in CONNECTORS-920.
Karl
On Wed, Apr 9, 2014 at 5:35 AM, Karl Wright <da...@gmail.com> wrote:
> Hi Hiroshi,
>
> I've created CONNECTORS-920 to describe this issue (
> https://issues.apache.org/jira/browse/CONNECTORS-920). In the future,
> for cases like this, you can create tickets yourself.
>
>
> One point, though, is that SocketTimeoutException is typically thrown when
> Solr fails to respond during the time a document is being posted to it.
> This is likely due to load on the Solr instance. Given that, it does not
> make much sense to skip the document, but to retry it later. This is the
> fix I will make.
>
> Karl
>
>
>
> On Tue, Apr 8, 2014 at 11:35 PM, Hiroshi Tatsumi <
> honekichi19@comet.ocn.ne.jp> wrote:
>
>> Hi,
>>
>> I'm using MCF1.5.1 and Solr4.6.1.
>>
>> When I use SolrConnecotor, sometimes SolrServerException occurs.
>>
>> Normally, SolrServerException is caught by HttpPoster, line 950.
>> But in my case, the inner exception of SolrServerException is
>> SocketTimeoutException, not SocketException.
>> So, activityCode is set to failed, then mcf interrupt the crawl process.
>>
>> In this case, I expect that mcf shouldn't interrupt the crawl process.
>> Mcf should skip the invalid file.
>> Could you modify the mcf code, or provide a option?
>>
>> If you could, I'm glad to being modified in the future version.
>>
>> Regards,
>> Hiroshi Tatsumi
>>
>>
>
Re: Handle SolrServerException in SolrConnecotor, HttpPoster
Posted by Karl Wright <da...@gmail.com>.
Hi Hiroshi,
I've created CONNECTORS-920 to describe this issue (
https://issues.apache.org/jira/browse/CONNECTORS-920). In the future, for
cases like this, you can create tickets yourself.
One point, though, is that SocketTimeoutException is typically thrown when
Solr fails to respond during the time a document is being posted to it.
This is likely due to load on the Solr instance. Given that, it does not
make much sense to skip the document, but to retry it later. This is the
fix I will make.
Karl
On Tue, Apr 8, 2014 at 11:35 PM, Hiroshi Tatsumi <
honekichi19@comet.ocn.ne.jp> wrote:
> Hi,
>
> I'm using MCF1.5.1 and Solr4.6.1.
>
> When I use SolrConnecotor, sometimes SolrServerException occurs.
>
> Normally, SolrServerException is caught by HttpPoster, line 950.
> But in my case, the inner exception of SolrServerException is
> SocketTimeoutException, not SocketException.
> So, activityCode is set to failed, then mcf interrupt the crawl process.
>
> In this case, I expect that mcf shouldn't interrupt the crawl process.
> Mcf should skip the invalid file.
> Could you modify the mcf code, or provide a option?
>
> If you could, I'm glad to being modified in the future version.
>
> Regards,
> Hiroshi Tatsumi
>
>