You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Christopher Gross <co...@gmail.com> on 2012/10/02 20:32:22 UTC
Nutch 2.1 fields
What fields are available to go in the solrindex-mapping.xml file for
Nutch 2.1? Is there a list somewhere?
In my 1.3 setup, I had "url" -- I don't think I added anything in like
a plugin to get that.
-- Chris
RE: Nutch 2.1 fields
Posted by j....@thomsonreuters.com.
Thanks Lewis that clears everything up.
>From looking at the code you linked to and the results in Solr my understanding is that primaryType and subType are not fields, they are additional entries in the type field. If that is correct it may be helpful to reflect that in "The Index Structure Page".
So in summary:
segment -- not relevant to Nutch 2.x
primaryType -- not a separate field but an additional entry in type
subType -- not a separate field but an additional entry in type
urlmeta -- not yet ported to Nutch 2.x
subcollection -- confirmed works fine with Nutch 2.x
-----Original Message-----
From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
Sent: Thursday, October 04, 2012 3:37 PM
To: user@nutch.apache.org
Subject: Re: Nutch 2.1 fields
[0] https://svn.apache.org/repos/asf/nutch/branches/2.x/src/plugin/index-more/src/java/org/apache/nutch/indexer/more/MoreIndexingFilter.java
On Thu, Oct 4, 2012 at 7:36 AM, Lewis John Mcgibbney <le...@gmail.com> wrote:
> Hi James,
>
> On Thu, Oct 4, 2012 at 2:59 AM, <j....@thomsonreuters.com> wrote:
>> Lewis and Chris,
>>
>> Agree that "The Index Structure" page is very useful documentation. I
>> went through the fields/plugins listed in your link using Nutch 2.1
>> rc and most work. I was able to get positive results for everything
>> except the following
>
> Great, thank you for the input this is extremely valuable.
>>
>> segment -- I am guessing this is not relevant to Nutch 2.x?
>
> you are correct
>
>> primaryType -- does this work with Nutch 2.x or do I need to look into my configuration?
>
> This should work out of the box *if* it is possible to index fields in
> such a manner. Please see the docs on the addType() method [0] for
> much more info on this one
>
>> subtype -- does this work Nutch 2.x or is my set up misconfigured?
>
> Please see above
>
>> urlmeta -- does this work Nutch 2.x or is my set up misconfigured?
>
> No you are correct. A port from Trunk to 2.x would be very welcome
> however
>
>> subcollection -- does this work with Nutch 2.x or is my set up misconfigured?
>
> Again this should work out of the box
>>
>> I notice the default installation of Nutch 2.x does not come with the urlmeta plugin nor the index-static plugin.
>
> Correct
>>
>> It might be useful to add a column to that page about which fields work with 1.x and which with 2.x. I think new users are finding it challenging to figure out which documentation applies to which version of Nutch.
>
> Correct. This is a great idea and I will set about it unless someone
> beats me to it as the wiki entry need another overhaul regardless.
>
> If anyone wishes and is able to contribute then please send me your
> wiki username and I can grant necessary Karma.
>
> Thank you
> Lewis
--
Lewis
Re: Nutch 2.1 fields
Posted by Lewis John Mcgibbney <le...@gmail.com>.
[0] https://svn.apache.org/repos/asf/nutch/branches/2.x/src/plugin/index-more/src/java/org/apache/nutch/indexer/more/MoreIndexingFilter.java
On Thu, Oct 4, 2012 at 7:36 AM, Lewis John Mcgibbney
<le...@gmail.com> wrote:
> Hi James,
>
> On Thu, Oct 4, 2012 at 2:59 AM, <j....@thomsonreuters.com> wrote:
>> Lewis and Chris,
>>
>> Agree that "The Index Structure" page is very useful documentation. I went through the fields/plugins listed in your link using Nutch 2.1 rc and most work. I was able to get positive results for everything except the following
>
> Great, thank you for the input this is extremely valuable.
>>
>> segment -- I am guessing this is not relevant to Nutch 2.x?
>
> you are correct
>
>> primaryType -- does this work with Nutch 2.x or do I need to look into my configuration?
>
> This should work out of the box *if* it is possible to index fields in
> such a manner. Please see the docs on the addType() method [0] for
> much more info on this one
>
>> subtype -- does this work Nutch 2.x or is my set up misconfigured?
>
> Please see above
>
>> urlmeta -- does this work Nutch 2.x or is my set up misconfigured?
>
> No you are correct. A port from Trunk to 2.x would be very welcome however
>
>> subcollection -- does this work with Nutch 2.x or is my set up misconfigured?
>
> Again this should work out of the box
>>
>> I notice the default installation of Nutch 2.x does not come with the urlmeta plugin nor the index-static plugin.
>
> Correct
>>
>> It might be useful to add a column to that page about which fields work with 1.x and which with 2.x. I think new users are finding it challenging to figure out which documentation applies to which version of Nutch.
>
> Correct. This is a great idea and I will set about it unless someone
> beats me to it as the wiki entry need another overhaul regardless.
>
> If anyone wishes and is able to contribute then please send me your
> wiki username and I can grant necessary Karma.
>
> Thank you
> Lewis
--
Lewis
Re: Nutch 2.1 fields
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi James,
On Thu, Oct 4, 2012 at 2:59 AM, <j....@thomsonreuters.com> wrote:
> Lewis and Chris,
>
> Agree that "The Index Structure" page is very useful documentation. I went through the fields/plugins listed in your link using Nutch 2.1 rc and most work. I was able to get positive results for everything except the following
Great, thank you for the input this is extremely valuable.
>
> segment -- I am guessing this is not relevant to Nutch 2.x?
you are correct
> primaryType -- does this work with Nutch 2.x or do I need to look into my configuration?
This should work out of the box *if* it is possible to index fields in
such a manner. Please see the docs on the addType() method [0] for
much more info on this one
> subtype -- does this work Nutch 2.x or is my set up misconfigured?
Please see above
> urlmeta -- does this work Nutch 2.x or is my set up misconfigured?
No you are correct. A port from Trunk to 2.x would be very welcome however
> subcollection -- does this work with Nutch 2.x or is my set up misconfigured?
Again this should work out of the box
>
> I notice the default installation of Nutch 2.x does not come with the urlmeta plugin nor the index-static plugin.
Correct
>
> It might be useful to add a column to that page about which fields work with 1.x and which with 2.x. I think new users are finding it challenging to figure out which documentation applies to which version of Nutch.
Correct. This is a great idea and I will set about it unless someone
beats me to it as the wiki entry need another overhaul regardless.
If anyone wishes and is able to contribute then please send me your
wiki username and I can grant necessary Karma.
Thank you
Lewis
RE: Nutch 2.1 fields
Posted by j....@thomsonreuters.com.
subcollection does work with 2.x and the problem was the configuration on my side (the subcollections.xml file in the conf folder).
So the list of fields in the "The Index Structure" page I can't confirm working with Nutch 2.x yet are:
segment
primaryType
subtype
urlmeta
-----Original Message-----
From: Sullivan, James (Financial&Risk)
Sent: Thursday, October 04, 2012 10:59 AM
To: user@nutch.apache.org
Subject: RE: Nutch 2.1 fields
Lewis and Chris,
Agree that "The Index Structure" page is very useful documentation. I went through the fields/plugins listed in your link using Nutch 2.1 rc and most work. I was able to get positive results for everything except the following
segment -- I am guessing this is not relevant to Nutch 2.x?
primaryType -- does this work with Nutch 2.x or do I need to look into my configuration?
subtype -- does this work Nutch 2.x or is my set up misconfigured?
urlmeta -- does this work Nutch 2.x or is my set up misconfigured?
subcollection -- does this work with Nutch 2.x or is my set up misconfigured?
I notice the default installation of Nutch 2.x does not come with the urlmeta plugin nor the index-static plugin.
It might be useful to add a column to that page about which fields work with 1.x and which with 2.x. I think new users are finding it challenging to figure out which documentation applies to which version of Nutch.
James
-----Original Message-----
From: Christopher Gross [mailto:cogross@gmail.com]
Sent: Wednesday, October 03, 2012 3:59 AM
To: user@nutch.apache.org
Subject: Re: Nutch 2.1 fields
Awesome, this is exactly what I needed. Thanks Lewis!
-- Chris
On Tue, Oct 2, 2012 at 2:50 PM, Lewis John Mcgibbney <le...@gmail.com> wrote:
> Hi Chris,
>
> Please see here [0] for the most up-to-date account of the fields for
> building your Solr index.
>
> I tried to bring this bang up to date a while back and more recently
> when writing some trivial plugin tests however please shout about
> anything which is not correct and we can edit accordingly.
>
> hth
>
> Lewis
>
> [0] http://wiki.apache.org/nutch/IndexStructure
>
> On Tue, Oct 2, 2012 at 7:32 PM, Christopher Gross <co...@gmail.com> wrote:
>> What fields are available to go in the solrindex-mapping.xml file for
>> Nutch 2.1? Is there a list somewhere?
>>
>> In my 1.3 setup, I had "url" -- I don't think I added anything in
>> like a plugin to get that.
>>
>> -- Chris
>
>
>
> --
> Lewis
RE: Nutch 2.1 fields
Posted by j....@thomsonreuters.com.
Lewis and Chris,
Agree that "The Index Structure" page is very useful documentation. I went through the fields/plugins listed in your link using Nutch 2.1 rc and most work. I was able to get positive results for everything except the following
segment -- I am guessing this is not relevant to Nutch 2.x?
primaryType -- does this work with Nutch 2.x or do I need to look into my configuration?
subtype -- does this work Nutch 2.x or is my set up misconfigured?
urlmeta -- does this work Nutch 2.x or is my set up misconfigured?
subcollection -- does this work with Nutch 2.x or is my set up misconfigured?
I notice the default installation of Nutch 2.x does not come with the urlmeta plugin nor the index-static plugin.
It might be useful to add a column to that page about which fields work with 1.x and which with 2.x. I think new users are finding it challenging to figure out which documentation applies to which version of Nutch.
James
-----Original Message-----
From: Christopher Gross [mailto:cogross@gmail.com]
Sent: Wednesday, October 03, 2012 3:59 AM
To: user@nutch.apache.org
Subject: Re: Nutch 2.1 fields
Awesome, this is exactly what I needed. Thanks Lewis!
-- Chris
On Tue, Oct 2, 2012 at 2:50 PM, Lewis John Mcgibbney <le...@gmail.com> wrote:
> Hi Chris,
>
> Please see here [0] for the most up-to-date account of the fields for
> building your Solr index.
>
> I tried to bring this bang up to date a while back and more recently
> when writing some trivial plugin tests however please shout about
> anything which is not correct and we can edit accordingly.
>
> hth
>
> Lewis
>
> [0] http://wiki.apache.org/nutch/IndexStructure
>
> On Tue, Oct 2, 2012 at 7:32 PM, Christopher Gross <co...@gmail.com> wrote:
>> What fields are available to go in the solrindex-mapping.xml file for
>> Nutch 2.1? Is there a list somewhere?
>>
>> In my 1.3 setup, I had "url" -- I don't think I added anything in
>> like a plugin to get that.
>>
>> -- Chris
>
>
>
> --
> Lewis
Re: Nutch 2.1 fields
Posted by Christopher Gross <co...@gmail.com>.
Awesome, this is exactly what I needed. Thanks Lewis!
-- Chris
On Tue, Oct 2, 2012 at 2:50 PM, Lewis John Mcgibbney
<le...@gmail.com> wrote:
> Hi Chris,
>
> Please see here [0] for the most up-to-date account of the fields for
> building your Solr index.
>
> I tried to bring this bang up to date a while back and more recently
> when writing some trivial plugin tests however please shout about
> anything which is not correct and we can edit accordingly.
>
> hth
>
> Lewis
>
> [0] http://wiki.apache.org/nutch/IndexStructure
>
> On Tue, Oct 2, 2012 at 7:32 PM, Christopher Gross <co...@gmail.com> wrote:
>> What fields are available to go in the solrindex-mapping.xml file for
>> Nutch 2.1? Is there a list somewhere?
>>
>> In my 1.3 setup, I had "url" -- I don't think I added anything in like
>> a plugin to get that.
>>
>> -- Chris
>
>
>
> --
> Lewis
Re: Nutch 2.1 fields
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Chris,
Please see here [0] for the most up-to-date account of the fields for
building your Solr index.
I tried to bring this bang up to date a while back and more recently
when writing some trivial plugin tests however please shout about
anything which is not correct and we can edit accordingly.
hth
Lewis
[0] http://wiki.apache.org/nutch/IndexStructure
On Tue, Oct 2, 2012 at 7:32 PM, Christopher Gross <co...@gmail.com> wrote:
> What fields are available to go in the solrindex-mapping.xml file for
> Nutch 2.1? Is there a list somewhere?
>
> In my 1.3 setup, I had "url" -- I don't think I added anything in like
> a plugin to get that.
>
> -- Chris
--
Lewis