You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Mike <mz...@gmail.com> on 2022/10/31 03:51:36 UTC
How should the headings plugin be configured?
Hello!
I've tried everything and set everything up and get the nutch headings
plugin working:
nutch-site.xml
<property>protocol-okhttp
<name>
<value>protocol-okhttp|...|parse-(html|tika|text|metatags)|index-(basic|anchor|more|metadata)|...|headings|nutch-extensionpoints</value>
</property>
schema.xml
<!-- fields for the headings plugin -->
<field name="h1" type="text_general" stored="true" indexed="true"
multiValued="true"/>
<field name="h2" type="text_general" stored="true" indexed="true"
multiValued="true"/>
<field name="h3" type="text_general" stored="true" indexed="true"
multiValued="true"/>
<field name="h4" type="text_general" stored="true" indexed="true"
multiValued="true"/>
<field name="h5" type="text_general" stored="true" indexed="true"
multiValued="true"/>
<field name="h6" type="text_general" stored="true" indexed="true"
multiValued="true"/>
index-writers.xml
<mapping>
<rename>
<field source="metatag.h1" dest="h1"/>
<field source="metatag.h2" dest="h2"/>
<field source="metatag.h3" dest="h3"/>
<field source="metatag.h4" dest="h4"/>
<field source="metatag.h5" dest="h5"/>
<field source="metatag.h6" dest="h6"/>
</rename>
...
After indexing to solr there are no HTML headings tags in my solr index,
what's missing?
thanks!
Re: How should the headings plugin be configured?
Posted by Markus Jelsma <ma...@openindex.io>.
Hello Mike,
I think it should be working just fine with it enabled in
protocol.includes. You can check Nutch' parser output by using:
$ bin/nutch parsechecker <URL>
You should see one or more h# output fields present. You can then use the
index-metadata plugin to map the parser output fields to the indexer output
by setting the values for index.parse.md.
Regards,
Markus
Op ma 31 okt. 2022 om 04:51 schreef Mike <mz...@gmail.com>:
> Hello!
>
> I've tried everything and set everything up and get the nutch headings
> plugin working:
>
> nutch-site.xml
>
> <property>protocol-okhttp
> <name>
>
>
> <value>protocol-okhttp|...|parse-(html|tika|text|metatags)|index-(basic|anchor|more|metadata)|...|headings|nutch-extensionpoints</value>
> </property>
>
> schema.xml
>
>
> <!-- fields for the headings plugin -->
> <field name="h1" type="text_general" stored="true" indexed="true"
> multiValued="true"/>
> <field name="h2" type="text_general" stored="true" indexed="true"
> multiValued="true"/>
> <field name="h3" type="text_general" stored="true" indexed="true"
> multiValued="true"/>
> <field name="h4" type="text_general" stored="true" indexed="true"
> multiValued="true"/>
> <field name="h5" type="text_general" stored="true" indexed="true"
> multiValued="true"/>
> <field name="h6" type="text_general" stored="true" indexed="true"
> multiValued="true"/>
>
> index-writers.xml
> <mapping>
> <rename>
> <field source="metatag.h1" dest="h1"/>
> <field source="metatag.h2" dest="h2"/>
> <field source="metatag.h3" dest="h3"/>
> <field source="metatag.h4" dest="h4"/>
> <field source="metatag.h5" dest="h5"/>
> <field source="metatag.h6" dest="h6"/>
> </rename>
> ...
>
> After indexing to solr there are no HTML headings tags in my solr index,
> what's missing?
>
> thanks!
>
Re: How should the headings plugin be configured?
Posted by Mike <mz...@gmail.com>.
Thanks Markus, I'm not sure where the error was but I reinstalled Nutch and
it works with your setup.
Am Mo., 31. Okt. 2022 um 14:36 Uhr schrieb Markus Jelsma <
markus.jelsma@openindex.io>:
> Hmmm, using a clean current Nutch i can get it to work with:
> <configuration>
> <property>
> <name>http.agent.name</name>
> <value>NutchTest</value>
> </property>
> <property>
> <name>index.parse.md</name>
> <value>h1,h2</value>
> </property>
> <property>
> <name>plugin.includes</name>
> <value>headings|protocol-http|parse-tika|index-metadata</value>
> </property>
> </configuration>
>
> $ bin/nutch indexchecker https://nutch.apache.org/
> digest : 13584e71e6e09a71071936feb97892b8
> h1 : Apache Nutchâ„¢
> id : https://nutch.apache.org/
>
> Can you check you configuration? Is a plugin name mispelled? Is the
> headings plugin active during fetch/parse? Is the index-metadata plugin
> active?
>
> Regards,
> Markus
>
>
> Op ma 31 okt. 2022 om 14:14 schreef Mike <mz...@gmail.com>:
>
> > Hello Markus!
> >
> > Thank you for taking care of my problem!
> >
> > I removed the metatag.h# fron index.parse.md but ntuch indexchecker do
> not
> > show me still the fields.
> >
> > Am Mo., 31. Okt. 2022 um 12:56 Uhr schrieb Markus Jelsma <
> > markus.jelsma@openindex.io>:
> >
> > > Hello Mike,
> > >
> > > Please remove the metatag.* prefix in the index.parse.md config and i
> > > think
> > > you should be fine.
> > >
> > > Regards,
> > > Markus
> > >
> > > Op ma 31 okt. 2022 om 12:32 schreef Mike <mz...@gmail.com>:
> > >
> > > > Yes, sorry, I also forgot to post this setting:
> > > >
> > > > <property>
> > > > <name>index.parse.md</name>
> > > >
> > > >
> > > >
> > >
> >
> <value>metatag.description,metatag.keywords,metatag.rating,metatag.h1,metatag.h2,metatag.h3,metatag.h4,metatag.h5,metatag.h6</value>
> > > > <description>
> > > > Comma-separated list of keys to be taken from the parse metadata
> to
> > > > generate fields.
> > > > Can be used e.g. for 'description' or 'keywords' provided that
> these
> > > > values are generated
> > > > by a parser (see parse-metatags plugin)
> > > > </description>
> > > > </property>
> > > >
> > > > The Nutch parsechecker shows me the fields but the indexchecker
> > doesn't.
> > > >
> > > > Am Mo., 31. Okt. 2022 um 04:51 Uhr schrieb Mike <mz579070@gmail.com
> >:
> > > >
> > > > > Hello!
> > > > >
> > > > > I've tried everything and set everything up and get the nutch
> > headings
> > > > > plugin working:
> > > > >
> > > > > nutch-site.xml
> > > > >
> > > > > <property>protocol-okhttp
> > > > > <name>
> > > > >
> > > > >
> > > >
> > >
> >
> <value>protocol-okhttp|...|parse-(html|tika|text|metatags)|index-(basic|anchor|more|metadata)|...|headings|nutch-extensionpoints</value>
> > > > > </property>
> > > > >
> > > > > schema.xml
> > > > >
> > > > >
> > > > > <!-- fields for the headings plugin -->
> > > > > <field name="h1" type="text_general" stored="true" indexed="true"
> > > > > multiValued="true"/>
> > > > > <field name="h2" type="text_general" stored="true" indexed="true"
> > > > > multiValued="true"/>
> > > > > <field name="h3" type="text_general" stored="true" indexed="true"
> > > > > multiValued="true"/>
> > > > > <field name="h4" type="text_general" stored="true" indexed="true"
> > > > > multiValued="true"/>
> > > > > <field name="h5" type="text_general" stored="true" indexed="true"
> > > > > multiValued="true"/>
> > > > > <field name="h6" type="text_general" stored="true" indexed="true"
> > > > > multiValued="true"/>
> > > > >
> > > > > index-writers.xml
> > > > > <mapping>
> > > > > <rename>
> > > > > <field source="metatag.h1" dest="h1"/>
> > > > > <field source="metatag.h2" dest="h2"/>
> > > > > <field source="metatag.h3" dest="h3"/>
> > > > > <field source="metatag.h4" dest="h4"/>
> > > > > <field source="metatag.h5" dest="h5"/>
> > > > > <field source="metatag.h6" dest="h6"/>
> > > > > </rename>
> > > > > ...
> > > > >
> > > > > After indexing to solr there are no HTML headings tags in my solr
> > > index,
> > > > > what's missing?
> > > > >
> > > > > thanks!
> > > > >
> > > >
> > >
> >
>
Re: How should the headings plugin be configured?
Posted by Markus Jelsma <ma...@openindex.io>.
Hmmm, using a clean current Nutch i can get it to work with:
<configuration>
<property>
<name>http.agent.name</name>
<value>NutchTest</value>
</property>
<property>
<name>index.parse.md</name>
<value>h1,h2</value>
</property>
<property>
<name>plugin.includes</name>
<value>headings|protocol-http|parse-tika|index-metadata</value>
</property>
</configuration>
$ bin/nutch indexchecker https://nutch.apache.org/
digest : 13584e71e6e09a71071936feb97892b8
h1 : Apache Nutchâ„¢
id : https://nutch.apache.org/
Can you check you configuration? Is a plugin name mispelled? Is the
headings plugin active during fetch/parse? Is the index-metadata plugin
active?
Regards,
Markus
Op ma 31 okt. 2022 om 14:14 schreef Mike <mz...@gmail.com>:
> Hello Markus!
>
> Thank you for taking care of my problem!
>
> I removed the metatag.h# fron index.parse.md but ntuch indexchecker do not
> show me still the fields.
>
> Am Mo., 31. Okt. 2022 um 12:56 Uhr schrieb Markus Jelsma <
> markus.jelsma@openindex.io>:
>
> > Hello Mike,
> >
> > Please remove the metatag.* prefix in the index.parse.md config and i
> > think
> > you should be fine.
> >
> > Regards,
> > Markus
> >
> > Op ma 31 okt. 2022 om 12:32 schreef Mike <mz...@gmail.com>:
> >
> > > Yes, sorry, I also forgot to post this setting:
> > >
> > > <property>
> > > <name>index.parse.md</name>
> > >
> > >
> > >
> >
> <value>metatag.description,metatag.keywords,metatag.rating,metatag.h1,metatag.h2,metatag.h3,metatag.h4,metatag.h5,metatag.h6</value>
> > > <description>
> > > Comma-separated list of keys to be taken from the parse metadata to
> > > generate fields.
> > > Can be used e.g. for 'description' or 'keywords' provided that these
> > > values are generated
> > > by a parser (see parse-metatags plugin)
> > > </description>
> > > </property>
> > >
> > > The Nutch parsechecker shows me the fields but the indexchecker
> doesn't.
> > >
> > > Am Mo., 31. Okt. 2022 um 04:51 Uhr schrieb Mike <mz...@gmail.com>:
> > >
> > > > Hello!
> > > >
> > > > I've tried everything and set everything up and get the nutch
> headings
> > > > plugin working:
> > > >
> > > > nutch-site.xml
> > > >
> > > > <property>protocol-okhttp
> > > > <name>
> > > >
> > > >
> > >
> >
> <value>protocol-okhttp|...|parse-(html|tika|text|metatags)|index-(basic|anchor|more|metadata)|...|headings|nutch-extensionpoints</value>
> > > > </property>
> > > >
> > > > schema.xml
> > > >
> > > >
> > > > <!-- fields for the headings plugin -->
> > > > <field name="h1" type="text_general" stored="true" indexed="true"
> > > > multiValued="true"/>
> > > > <field name="h2" type="text_general" stored="true" indexed="true"
> > > > multiValued="true"/>
> > > > <field name="h3" type="text_general" stored="true" indexed="true"
> > > > multiValued="true"/>
> > > > <field name="h4" type="text_general" stored="true" indexed="true"
> > > > multiValued="true"/>
> > > > <field name="h5" type="text_general" stored="true" indexed="true"
> > > > multiValued="true"/>
> > > > <field name="h6" type="text_general" stored="true" indexed="true"
> > > > multiValued="true"/>
> > > >
> > > > index-writers.xml
> > > > <mapping>
> > > > <rename>
> > > > <field source="metatag.h1" dest="h1"/>
> > > > <field source="metatag.h2" dest="h2"/>
> > > > <field source="metatag.h3" dest="h3"/>
> > > > <field source="metatag.h4" dest="h4"/>
> > > > <field source="metatag.h5" dest="h5"/>
> > > > <field source="metatag.h6" dest="h6"/>
> > > > </rename>
> > > > ...
> > > >
> > > > After indexing to solr there are no HTML headings tags in my solr
> > index,
> > > > what's missing?
> > > >
> > > > thanks!
> > > >
> > >
> >
>
Re: How should the headings plugin be configured?
Posted by Mike <mz...@gmail.com>.
Hello Markus!
Thank you for taking care of my problem!
I removed the metatag.h# fron index.parse.md but ntuch indexchecker do not
show me still the fields.
Am Mo., 31. Okt. 2022 um 12:56 Uhr schrieb Markus Jelsma <
markus.jelsma@openindex.io>:
> Hello Mike,
>
> Please remove the metatag.* prefix in the index.parse.md config and i
> think
> you should be fine.
>
> Regards,
> Markus
>
> Op ma 31 okt. 2022 om 12:32 schreef Mike <mz...@gmail.com>:
>
> > Yes, sorry, I also forgot to post this setting:
> >
> > <property>
> > <name>index.parse.md</name>
> >
> >
> >
> <value>metatag.description,metatag.keywords,metatag.rating,metatag.h1,metatag.h2,metatag.h3,metatag.h4,metatag.h5,metatag.h6</value>
> > <description>
> > Comma-separated list of keys to be taken from the parse metadata to
> > generate fields.
> > Can be used e.g. for 'description' or 'keywords' provided that these
> > values are generated
> > by a parser (see parse-metatags plugin)
> > </description>
> > </property>
> >
> > The Nutch parsechecker shows me the fields but the indexchecker doesn't.
> >
> > Am Mo., 31. Okt. 2022 um 04:51 Uhr schrieb Mike <mz...@gmail.com>:
> >
> > > Hello!
> > >
> > > I've tried everything and set everything up and get the nutch headings
> > > plugin working:
> > >
> > > nutch-site.xml
> > >
> > > <property>protocol-okhttp
> > > <name>
> > >
> > >
> >
> <value>protocol-okhttp|...|parse-(html|tika|text|metatags)|index-(basic|anchor|more|metadata)|...|headings|nutch-extensionpoints</value>
> > > </property>
> > >
> > > schema.xml
> > >
> > >
> > > <!-- fields for the headings plugin -->
> > > <field name="h1" type="text_general" stored="true" indexed="true"
> > > multiValued="true"/>
> > > <field name="h2" type="text_general" stored="true" indexed="true"
> > > multiValued="true"/>
> > > <field name="h3" type="text_general" stored="true" indexed="true"
> > > multiValued="true"/>
> > > <field name="h4" type="text_general" stored="true" indexed="true"
> > > multiValued="true"/>
> > > <field name="h5" type="text_general" stored="true" indexed="true"
> > > multiValued="true"/>
> > > <field name="h6" type="text_general" stored="true" indexed="true"
> > > multiValued="true"/>
> > >
> > > index-writers.xml
> > > <mapping>
> > > <rename>
> > > <field source="metatag.h1" dest="h1"/>
> > > <field source="metatag.h2" dest="h2"/>
> > > <field source="metatag.h3" dest="h3"/>
> > > <field source="metatag.h4" dest="h4"/>
> > > <field source="metatag.h5" dest="h5"/>
> > > <field source="metatag.h6" dest="h6"/>
> > > </rename>
> > > ...
> > >
> > > After indexing to solr there are no HTML headings tags in my solr
> index,
> > > what's missing?
> > >
> > > thanks!
> > >
> >
>
Re: How should the headings plugin be configured?
Posted by Markus Jelsma <ma...@openindex.io>.
Hello Mike,
Please remove the metatag.* prefix in the index.parse.md config and i think
you should be fine.
Regards,
Markus
Op ma 31 okt. 2022 om 12:32 schreef Mike <mz...@gmail.com>:
> Yes, sorry, I also forgot to post this setting:
>
> <property>
> <name>index.parse.md</name>
>
>
> <value>metatag.description,metatag.keywords,metatag.rating,metatag.h1,metatag.h2,metatag.h3,metatag.h4,metatag.h5,metatag.h6</value>
> <description>
> Comma-separated list of keys to be taken from the parse metadata to
> generate fields.
> Can be used e.g. for 'description' or 'keywords' provided that these
> values are generated
> by a parser (see parse-metatags plugin)
> </description>
> </property>
>
> The Nutch parsechecker shows me the fields but the indexchecker doesn't.
>
> Am Mo., 31. Okt. 2022 um 04:51 Uhr schrieb Mike <mz...@gmail.com>:
>
> > Hello!
> >
> > I've tried everything and set everything up and get the nutch headings
> > plugin working:
> >
> > nutch-site.xml
> >
> > <property>protocol-okhttp
> > <name>
> >
> >
> <value>protocol-okhttp|...|parse-(html|tika|text|metatags)|index-(basic|anchor|more|metadata)|...|headings|nutch-extensionpoints</value>
> > </property>
> >
> > schema.xml
> >
> >
> > <!-- fields for the headings plugin -->
> > <field name="h1" type="text_general" stored="true" indexed="true"
> > multiValued="true"/>
> > <field name="h2" type="text_general" stored="true" indexed="true"
> > multiValued="true"/>
> > <field name="h3" type="text_general" stored="true" indexed="true"
> > multiValued="true"/>
> > <field name="h4" type="text_general" stored="true" indexed="true"
> > multiValued="true"/>
> > <field name="h5" type="text_general" stored="true" indexed="true"
> > multiValued="true"/>
> > <field name="h6" type="text_general" stored="true" indexed="true"
> > multiValued="true"/>
> >
> > index-writers.xml
> > <mapping>
> > <rename>
> > <field source="metatag.h1" dest="h1"/>
> > <field source="metatag.h2" dest="h2"/>
> > <field source="metatag.h3" dest="h3"/>
> > <field source="metatag.h4" dest="h4"/>
> > <field source="metatag.h5" dest="h5"/>
> > <field source="metatag.h6" dest="h6"/>
> > </rename>
> > ...
> >
> > After indexing to solr there are no HTML headings tags in my solr index,
> > what's missing?
> >
> > thanks!
> >
>
Re: How should the headings plugin be configured?
Posted by Mike <mz...@gmail.com>.
Yes, sorry, I also forgot to post this setting:
<property>
<name>index.parse.md</name>
<value>metatag.description,metatag.keywords,metatag.rating,metatag.h1,metatag.h2,metatag.h3,metatag.h4,metatag.h5,metatag.h6</value>
<description>
Comma-separated list of keys to be taken from the parse metadata to
generate fields.
Can be used e.g. for 'description' or 'keywords' provided that these
values are generated
by a parser (see parse-metatags plugin)
</description>
</property>
The Nutch parsechecker shows me the fields but the indexchecker doesn't.
Am Mo., 31. Okt. 2022 um 04:51 Uhr schrieb Mike <mz...@gmail.com>:
> Hello!
>
> I've tried everything and set everything up and get the nutch headings
> plugin working:
>
> nutch-site.xml
>
> <property>protocol-okhttp
> <name>
>
> <value>protocol-okhttp|...|parse-(html|tika|text|metatags)|index-(basic|anchor|more|metadata)|...|headings|nutch-extensionpoints</value>
> </property>
>
> schema.xml
>
>
> <!-- fields for the headings plugin -->
> <field name="h1" type="text_general" stored="true" indexed="true"
> multiValued="true"/>
> <field name="h2" type="text_general" stored="true" indexed="true"
> multiValued="true"/>
> <field name="h3" type="text_general" stored="true" indexed="true"
> multiValued="true"/>
> <field name="h4" type="text_general" stored="true" indexed="true"
> multiValued="true"/>
> <field name="h5" type="text_general" stored="true" indexed="true"
> multiValued="true"/>
> <field name="h6" type="text_general" stored="true" indexed="true"
> multiValued="true"/>
>
> index-writers.xml
> <mapping>
> <rename>
> <field source="metatag.h1" dest="h1"/>
> <field source="metatag.h2" dest="h2"/>
> <field source="metatag.h3" dest="h3"/>
> <field source="metatag.h4" dest="h4"/>
> <field source="metatag.h5" dest="h5"/>
> <field source="metatag.h6" dest="h6"/>
> </rename>
> ...
>
> After indexing to solr there are no HTML headings tags in my solr index,
> what's missing?
>
> thanks!
>