You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Yusniel Hidalgo Delgado <yh...@uci.cu> on 2015/02/01 16:34:32 UTC
How to implement an own crawler for specific tasks with nutch?
Hi community,
I am working on a project to develop a focused crawler. More specifically, we need to crawl several websites and download all PDF files from them. My question is:
It is possible to implement a focused crawler using nutch as API for own java application?
I am searching in the web and the nutch list, but I can't find nothing about it.
Thanks in advance.
Best regards.
Yusniel Hidalgo Delgado
Semantic Web Research Group
University of Informatics Sciences
http://gws-uci.blogspot.com/
Havana, Cuba
---------------------------------------------------
XII Aniversario de la creación de la Universidad de las Ciencias Informáticas. 12 años de historia junto a Fidel. 12 de diciembre de 2014.
Re: [MASSMAIL]Re: How to implement an own crawler for specific
tasks with nutch?
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Dear Yusniel,
Great question! Yes, in Nutch 2.x it is, there are REST service
APIs:
https://wiki.apache.org/nutch/NutchRESTAPI
I and members of my team are working on porting these
services to the 1.x branch using Apache CXF instead of the
Restlet technology:
https://issues.apache.org/jira/browse/NUTCH-1916
We have one example that is already dependent on this, which
is the Nutch 2 Wicket based application (that we would like
in Nutch 1.x as well):
https://issues.apache.org/jira/browse/NUTCH-841
There is also an effort within the Python community led by
Continuum Analytics and our team on the DARPA Memex project
to construct a nutchpy application which would also have
a dependency on this:
https://github.com/ContinuumIO/nutchpy
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Yusniel Hidalgo Delgado <yh...@uci.cu>
Reply-To: "user@nutch.apache.org" <us...@nutch.apache.org>
Date: Sunday, February 1, 2015 at 9:27 AM
To: "user@nutch.apache.org" <us...@nutch.apache.org>
Subject: Re: [MASSMAIL]Re: How to implement an own crawler for specific
tasks with nutch?
>Hi Chris,
>
>Thank you very much for your comments, they are very helpful. It is
>exactly what we need. However, as I can see, Nutch works as a standalone
>application. I was wondering if it is possible to use just the crawl
>component of Nutch in a java application? Is there an API of Nutch?
>
>Cheers,
>
>Yusniel Hidalgo Delgado
>Semantic Web Research Group
>University of Informatics Sciences
>http://gws-uci.blogspot.com/
>Havana, Cuba
>
>----- Mensaje original -----
>De: "Chris A Mattmann (3980)" <ch...@jpl.nasa.gov>
>Para: user@nutch.apache.org
>Enviados: Domingo, 1 de Febrero 2015 11:44:45
>Asunto: [MASSMAIL]Re: How to implement an own crawler for specific tasks
>with nutch?
>
>Hi Yusniel,
>
>Thanks for your question and for using Nutch!
>
>Yep it’s possible to implement a focused crawler,
>which is defined hopefully by the following criteria:
>
>1. partitioned URL space (in Nutch you use URL filters
>and normalizers for this and seed lists and injection)
>2. only certain content types (in Nutch you configure
>Apache Tika here via the parse-tika plugin)
>3. only certain metadata needed (here you use custom
>index-* plugins, along with Solr configuration or
>ElasticSearch config to control the flow of metadata
>into the index)
>
>Are there other requirements that you have for focused
>crawling beyond that?
>
>Cheers,
>Chris
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Chief Architect
>Instrument Software and Science Data Systems Section (398)
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 168-519, Mailstop: 168-527
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Associate Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
>-----Original Message-----
>From: Yusniel Hidalgo Delgado <yh...@uci.cu>
>Reply-To: "user@nutch.apache.org" <us...@nutch.apache.org>
>Date: Sunday, February 1, 2015 at 7:34 AM
>To: "user@nutch.apache.org" <us...@nutch.apache.org>
>Subject: How to implement an own crawler for specific tasks with nutch?
>
>>Hi community,
>>I am working on a project to develop a focused crawler. More
>>specifically, we need to crawl several websites and download all PDF
>>files from them. My question is:
>>It is possible to implement a focused crawler using nutch as API for own
>>java application?
>>I am searching in the web and the nutch list, but I can't find nothing
>>about it.
>>
>>Thanks in advance.
>>
>>Best regards.
>>
>>Yusniel Hidalgo Delgado
>>Semantic Web Research Group
>>University of Informatics Sciences
>>http://gws-uci.blogspot.com/
>>Havana, Cuba
>>
>>
>>
>>
>>---------------------------------------------------
>>XII Aniversario de la creación de la Universidad de las Ciencias
>>Informáticas. 12 años de historia junto a Fidel. 12 de diciembre de 2014.
>
>
>
>---------------------------------------------------
>XII Aniversario de la creación de la Universidad de las Ciencias
>Informáticas. 12 años de historia junto a Fidel. 12 de diciembre de 2014.
>
Re: [MASSMAIL]Re: How to implement an own crawler for specific
tasks with nutch?
Posted by Yusniel Hidalgo Delgado <yh...@uci.cu>.
Hi Chris,
Thank you very much for your comments, they are very helpful. It is exactly what we need. However, as I can see, Nutch works as a standalone application. I was wondering if it is possible to use just the crawl component of Nutch in a java application? Is there an API of Nutch?
Cheers,
Yusniel Hidalgo Delgado
Semantic Web Research Group
University of Informatics Sciences
http://gws-uci.blogspot.com/
Havana, Cuba
----- Mensaje original -----
De: "Chris A Mattmann (3980)" <ch...@jpl.nasa.gov>
Para: user@nutch.apache.org
Enviados: Domingo, 1 de Febrero 2015 11:44:45
Asunto: [MASSMAIL]Re: How to implement an own crawler for specific tasks with nutch?
Hi Yusniel,
Thanks for your question and for using Nutch!
Yep it’s possible to implement a focused crawler,
which is defined hopefully by the following criteria:
1. partitioned URL space (in Nutch you use URL filters
and normalizers for this and seed lists and injection)
2. only certain content types (in Nutch you configure
Apache Tika here via the parse-tika plugin)
3. only certain metadata needed (here you use custom
index-* plugins, along with Solr configuration or
ElasticSearch config to control the flow of metadata
into the index)
Are there other requirements that you have for focused
crawling beyond that?
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Yusniel Hidalgo Delgado <yh...@uci.cu>
Reply-To: "user@nutch.apache.org" <us...@nutch.apache.org>
Date: Sunday, February 1, 2015 at 7:34 AM
To: "user@nutch.apache.org" <us...@nutch.apache.org>
Subject: How to implement an own crawler for specific tasks with nutch?
>Hi community,
>I am working on a project to develop a focused crawler. More
>specifically, we need to crawl several websites and download all PDF
>files from them. My question is:
>It is possible to implement a focused crawler using nutch as API for own
>java application?
>I am searching in the web and the nutch list, but I can't find nothing
>about it.
>
>Thanks in advance.
>
>Best regards.
>
>Yusniel Hidalgo Delgado
>Semantic Web Research Group
>University of Informatics Sciences
>http://gws-uci.blogspot.com/
>Havana, Cuba
>
>
>
>
>---------------------------------------------------
>XII Aniversario de la creación de la Universidad de las Ciencias
>Informáticas. 12 años de historia junto a Fidel. 12 de diciembre de 2014.
---------------------------------------------------
XII Aniversario de la creación de la Universidad de las Ciencias Informáticas. 12 años de historia junto a Fidel. 12 de diciembre de 2014.
Re: How to implement an own crawler for specific tasks with nutch?
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Hi Yusniel,
Thanks for your question and for using Nutch!
Yep it’s possible to implement a focused crawler,
which is defined hopefully by the following criteria:
1. partitioned URL space (in Nutch you use URL filters
and normalizers for this and seed lists and injection)
2. only certain content types (in Nutch you configure
Apache Tika here via the parse-tika plugin)
3. only certain metadata needed (here you use custom
index-* plugins, along with Solr configuration or
ElasticSearch config to control the flow of metadata
into the index)
Are there other requirements that you have for focused
crawling beyond that?
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Yusniel Hidalgo Delgado <yh...@uci.cu>
Reply-To: "user@nutch.apache.org" <us...@nutch.apache.org>
Date: Sunday, February 1, 2015 at 7:34 AM
To: "user@nutch.apache.org" <us...@nutch.apache.org>
Subject: How to implement an own crawler for specific tasks with nutch?
>Hi community,
>I am working on a project to develop a focused crawler. More
>specifically, we need to crawl several websites and download all PDF
>files from them. My question is:
>It is possible to implement a focused crawler using nutch as API for own
>java application?
>I am searching in the web and the nutch list, but I can't find nothing
>about it.
>
>Thanks in advance.
>
>Best regards.
>
>Yusniel Hidalgo Delgado
>Semantic Web Research Group
>University of Informatics Sciences
>http://gws-uci.blogspot.com/
>Havana, Cuba
>
>
>
>
>---------------------------------------------------
>XII Aniversario de la creación de la Universidad de las Ciencias
>Informáticas. 12 años de historia junto a Fidel. 12 de diciembre de 2014.