You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by sw...@quandatics.com on 2021/10/22 08:10:30 UTC
Cant integrate the kerberos enabled solr cloud with nutch
Hi,
We have encountered a problem which can't integrate the kerberos enabled
solr cloud with nutch.
When execute "nutch index crawl/crawldb/ -linkdb crawl/linkdb/ $s1 -filter
-normalize" command ,it will fail with "HTTP ERROR 401Problem accessing
/solr/admin/collections. Reason:Authentication required" but we able to curl
it with the keytab.
Version of Nutch :1.18
Your Sincerely,
Shi Wei
Re: Cant integrate the kerberos enabled solr cloud with nutch
Posted by Wei <sw...@quandatics.com>.
Hi Sebastian,
Thanks for your reply.
We understand that the current indexer-solr plugin only supports the basics authentication. We will open the jira issue accordingly. Usually how long it would take for the jira to resolve?
Besides, is there any other workaround that we could try for the described issue? It's because upon our research in these few days, it seems that somebody were able to connect to the Solr with Kerberos authentication via the NTLM scheme of HttpAuthenticationSchemes. Could you assist to check whether this would work?
https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=115512111#content/view/115512111 <https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=115512111#content/view/115512111>
Your Sincerely,
Shi Wei
> On 22 Oct 2021, at 6:55 PM, Sebastian Nagel <wa...@googlemail.com.INVALID> wrote:
>
> Hi Shi Wei,
>
> > kerberos
>
> sorry, I missed this detail. The plugin indexer-solr for now
> only supports basic authentication.
>
> Could you open a Jira issue to get Kerberos authentication
> implemented on the Nutch site?
> https://issues.apache.org/jira/projects/NUTCH
>
> See also:
> https://solr.apache.org/guide/8_5/kerberos-authentication-plugin.html#using-solrj-with-a-kerberized-solr
>
> Thanks,
> Sebastian
>
> On 10/22/21 12:01 PM, sw.ling@quandatics.com wrote:
>> Hi Sebastian,
>> Here is the index-writers.xml you requested. Thank
>> Your Sincerely,
>> Shi Wei
>> -----Original Message-----
>> From: Sebastian Nagel <wa...@googlemail.com.INVALID>
>> Sent: Friday, 22 October, 2021 5:46 PM
>> To: user@nutch.apache.org
>> Subject: Re: Cant integrate the kerberos enabled solr cloud with nutch
>> Hi Shi Wei,
>> could you also share the index writer configuration (conf/index-writers.xml)?
>> The default is unauthenticated access to Solr, see the snippet below.
>> The file httpclient-auth.xml is not relevant for the Solr indexer, it's used if a crawled web site requires authentication in order to fetch the content via the plugin protocol-httpclient.
>> Best,
>> Sebastian
>> <writer id="indexer_solr_1" class="org.apache.nutch.indexwriter.solr.SolrIndexWriter">
>> <parameters>
>> <param name="type" value="http"/>
>> <param name="url" value="http://localhost:8983/solr/nutch"/>
>> <param name="collection" value=""/>
>> <param name="weight.field" value=""/>
>> <param name="commitSize" value="1000"/>
>> <param name="auth" value="false"/>
>> <param name="username" value="username"/>
>> <param name="password" value="password"/>
>> On 10/22/21 10:10 AM, sw.ling@quandatics.com wrote:
>>> Hi,
>>>
>>> We have encountered a problem which can’t integrate the kerberos enabled solr cloud with nutch.
>>>
>>> When execute "nutch index crawl/crawldb/ -linkdb crawl/linkdb/ $s1
>>> -filter -normalize" command ,it will fail with "HTTP ERROR 401Problem accessing /solr/admin/collections. Reason:Authentication required" but we able to curl it with the keytab.
>>>
>>> Version of Nutch :1.18
>>>
>>> Your Sincerely,
>>>
>>> Shi Wei
>>>
>
Re: Cant integrate the kerberos enabled solr cloud with nutch
Posted by Sebastian Nagel <wa...@googlemail.com.INVALID>.
Hi Shi Wei,
> kerberos
sorry, I missed this detail. The plugin indexer-solr for now
only supports basic authentication.
Could you open a Jira issue to get Kerberos authentication
implemented on the Nutch site?
https://issues.apache.org/jira/projects/NUTCH
See also:
https://solr.apache.org/guide/8_5/kerberos-authentication-plugin.html#using-solrj-with-a-kerberized-solr
Thanks,
Sebastian
On 10/22/21 12:01 PM, sw.ling@quandatics.com wrote:
> Hi Sebastian,
>
> Here is the index-writers.xml you requested. Thank
>
> Your Sincerely,
> Shi Wei
>
>
> -----Original Message-----
> From: Sebastian Nagel <wa...@googlemail.com.INVALID>
> Sent: Friday, 22 October, 2021 5:46 PM
> To: user@nutch.apache.org
> Subject: Re: Cant integrate the kerberos enabled solr cloud with nutch
>
> Hi Shi Wei,
>
> could you also share the index writer configuration (conf/index-writers.xml)?
>
> The default is unauthenticated access to Solr, see the snippet below.
> The file httpclient-auth.xml is not relevant for the Solr indexer, it's used if a crawled web site requires authentication in order to fetch the content via the plugin protocol-httpclient.
>
> Best,
> Sebastian
>
> <writer id="indexer_solr_1" class="org.apache.nutch.indexwriter.solr.SolrIndexWriter">
> <parameters>
> <param name="type" value="http"/>
> <param name="url" value="http://localhost:8983/solr/nutch"/>
> <param name="collection" value=""/>
> <param name="weight.field" value=""/>
> <param name="commitSize" value="1000"/>
> <param name="auth" value="false"/>
> <param name="username" value="username"/>
> <param name="password" value="password"/>
>
>
> On 10/22/21 10:10 AM, sw.ling@quandatics.com wrote:
>> Hi,
>>
>> We have encountered a problem which can’t integrate the kerberos enabled solr cloud with nutch.
>>
>> When execute "nutch index crawl/crawldb/ -linkdb crawl/linkdb/ $s1
>> -filter -normalize" command ,it will fail with "HTTP ERROR 401Problem accessing /solr/admin/collections. Reason:Authentication required" but we able to curl it with the keytab.
>>
>> Version of Nutch :1.18
>>
>> Your Sincerely,
>>
>> Shi Wei
>>
RE: Cant integrate the kerberos enabled solr cloud with nutch
Posted by sw...@quandatics.com.
Hi Sebastian,
Here is the index-writers.xml you requested. Thank
Your Sincerely,
Shi Wei
-----Original Message-----
From: Sebastian Nagel <wa...@googlemail.com.INVALID>
Sent: Friday, 22 October, 2021 5:46 PM
To: user@nutch.apache.org
Subject: Re: Cant integrate the kerberos enabled solr cloud with nutch
Hi Shi Wei,
could you also share the index writer configuration (conf/index-writers.xml)?
The default is unauthenticated access to Solr, see the snippet below.
The file httpclient-auth.xml is not relevant for the Solr indexer, it's used if a crawled web site requires authentication in order to fetch the content via the plugin protocol-httpclient.
Best,
Sebastian
<writer id="indexer_solr_1" class="org.apache.nutch.indexwriter.solr.SolrIndexWriter">
<parameters>
<param name="type" value="http"/>
<param name="url" value="http://localhost:8983/solr/nutch"/>
<param name="collection" value=""/>
<param name="weight.field" value=""/>
<param name="commitSize" value="1000"/>
<param name="auth" value="false"/>
<param name="username" value="username"/>
<param name="password" value="password"/>
On 10/22/21 10:10 AM, sw.ling@quandatics.com wrote:
> Hi,
>
> We have encountered a problem which can’t integrate the kerberos enabled solr cloud with nutch.
>
> When execute "nutch index crawl/crawldb/ -linkdb crawl/linkdb/ $s1
> -filter -normalize" command ,it will fail with "HTTP ERROR 401Problem accessing /solr/admin/collections. Reason:Authentication required" but we able to curl it with the keytab.
>
> Version of Nutch :1.18
>
> Your Sincerely,
>
> Shi Wei
>
Re: Cant integrate the kerberos enabled solr cloud with nutch
Posted by Sebastian Nagel <wa...@googlemail.com.INVALID>.
Hi Shi Wei,
could you also share the index writer configuration (conf/index-writers.xml)?
The default is unauthenticated access to Solr, see the snippet below.
The file httpclient-auth.xml is not relevant for the Solr indexer, it's
used if a crawled web site requires authentication in order to fetch
the content via the plugin protocol-httpclient.
Best,
Sebastian
<writer id="indexer_solr_1" class="org.apache.nutch.indexwriter.solr.SolrIndexWriter">
<parameters>
<param name="type" value="http"/>
<param name="url" value="http://localhost:8983/solr/nutch"/>
<param name="collection" value=""/>
<param name="weight.field" value=""/>
<param name="commitSize" value="1000"/>
<param name="auth" value="false"/>
<param name="username" value="username"/>
<param name="password" value="password"/>
On 10/22/21 10:10 AM, sw.ling@quandatics.com wrote:
> Hi,
>
> We have encountered a problem which can’t integrate the kerberos enabled solr cloud with nutch.
>
> When execute "nutch index crawl/crawldb/ -linkdb crawl/linkdb/ $s1 -filter -normalize" command ,it will fail with "HTTP ERROR 401Problem
> accessing /solr/admin/collections. Reason:Authentication required" but we able to curl it with the keytab.
>
> Version of Nutch :1.18
>
> Your Sincerely,
>
> Shi Wei
>