You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Dominique Bejean <do...@eolya.fr> on 2011/03/02 01:25:08 UTC
[ANNOUNCE] Web Crawler
Hi,
I would like to announce Crawl Anywhere. Crawl-Anywhere is a Java Web
Crawler. It includes :
* a crawler
* a document processing pipeline
* a solr indexer
The crawler has a web administration in order to manage web sites to be
crawled. Each web site crawl is configured with a lot of possible
parameters (no all mandatory) :
* number of simultaneous items crawled by site
* recrawl period rules based on item type (html, PDF, …)
* item type inclusion / exclusion rules
* item path inclusion / exclusion / strategy rules
* max depth
* web site authentication
* language
* country
* tags
* collections
* ...
The pileline includes various ready to use stages (text extraction,
language detection, Solr ready to index xml writer, ...).
All is very configurable and extendible either by scripting or java coding.
With scripting technology, you can help the crawler to handle javascript
links or help the pipeline to extract relevant title and cleanup the
html pages (remove menus, header, footers, ..)
With java coding, you can develop your own pipeline stage stage
The Crawl Anywhere web site provides good explanations and screen shots.
All is documented in a wiki.
The current version is 1.1.4. You can download and try it out from here
: www.crawl-anywhere.com
Regards
Dominique
Re: [ANNOUNCE] Web Crawler
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
I did see this message (again). Please, use the new dedicated
Crawl-Anywhere forum for your next questions.
https://groups.google.com/forum/#!forum/crawl-anywhere
Did you solve your problem ?
Thank you
Dominique
Le 29/01/13 09:28, SivaKarthik a écrit :
> Hi,
> i resolved the issue "Access denied for user 'crawler'@'localhost' (using
> password: YES)"
>
> mysql user crawler/crawler was created and privileges added as mentioned in
> the tutorial..
> Thank you.
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p4036978.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Dominique Béjean
+33 6 08 46 12 43
skype: dbejean
www.eolya.fr
www.crawl-anywhere.com
www.mysolrserver.com
Re: [ANNOUNCE] Web Crawler
Posted by SivaKarthik <si...@gmail.com>.
Hi,
i resolved the issue "Access denied for user 'crawler'@'localhost' (using
password: YES)"
mysql user crawler/crawler was created and privileges added as mentioned in
the tutorial..
Thank you.
--
View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p4036978.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: [ANNOUNCE] Web Crawler
Posted by SivaKarthik <si...@gmail.com>.
Klein,
Thank you for ur reply..
i hosted the application in apache2 server
and able to access the link http://localhost/search/
but while accessing http://localhost/crawler/login.php
its showing the error msg as
"Access denied for user 'crawler'@'localhost' (using
password: YES)"
i tried to access
http://localhost/crawler/log.php
http://localhost/crawler/display.php
but all throws the same error msg
"Access denied for user 'crawler'@'localhost' (using
password: YES)"
for testing purpose
i created test1.html and test2.php under /opt/crawler/web/crawler/pub
folder
and i succeeded to access it
http://localhost/crawler/test2.php
http://localhost/crawler/test1.html
im not completely sure why the Access denied error for login.php page
Any idea?
Regards
--
View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p4036966.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: [ANNOUNCE] Web Crawler
Posted by "O. Klein" <kl...@octoweb.nl>.
This is actualy showing it works.
crawlerws is used by Crawl Anywhere UI and will pass it the correct
arguments when needed.
SivaKarthik wrote
> Hii,
> I'm trying to configure crawl-anywhere 3.0.3 version in my local system..
> i'm following the steps from the page
> http://www.crawl-anywhere.com/installation-v300/
> but, crawlerws is failing and throwing the below error message in the
> brower
> http://localhost:8080/crawlerws/
> <error>
>
> <errno>
> 1
> </errno>
>
> <errmsg>
> Missing action
> </errmsg>
> </error>
> Not sure where im doing wrong.. could please help me out to resolve the
> problem.. thank you.
--
View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p4036520.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: [ANNOUNCE] Web Crawler
Posted by SivaKarthik <si...@gmail.com>.
Hii,
I'm trying to configure crawl-anywhere 3.0.3 version in my local system..
i'm following the steps from the page
http://www.crawl-anywhere.com/installation-v300/
but, crawlerws is failing and throwing the below error message in the
brower
http://localhost:8080/crawlerws/
<error>
<errno>1</errno>
<errmsg>Missing action</errmsg>
</error>
Not sure where im doing wrong.. could please help me out to resolve the
problem.. thank you.
--
View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p4036493.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: [ANNOUNCE] Web Crawler
Posted by Dominique Bejean <do...@eolya.fr>.
Lukas,
I am thinking about it but no decision yet.
Anyway, in next release, I will provide source code of pipeline stages
and connectors as samples.
Dominique
Le 02/03/11 10:01, Lukáš Vlček a écrit :
> Hi,
>
> is there any plan to open source it?
>
> Regards,
> Lukas
>
> [OT] I tried HuriSearch, input "Java" into search field, it returned a
> lot of references to coldfusion error pages. May be a recrawl would help?
>
> On Wed, Mar 2, 2011 at 1:25 AM, Dominique Bejean
> <dominique.bejean@eolya.fr <ma...@eolya.fr>> wrote:
>
> Hi,
>
> I would like to announce Crawl Anywhere. Crawl-Anywhere is a Java
> Web Crawler. It includes :
>
> * a crawler
> * a document processing pipeline
> * a solr indexer
>
> The crawler has a web administration in order to manage web sites
> to be crawled. Each web site crawl is configured with a lot of
> possible parameters (no all mandatory) :
>
> * number of simultaneous items crawled by site
> * recrawl period rules based on item type (html, PDF, …)
> * item type inclusion / exclusion rules
> * item path inclusion / exclusion / strategy rules
> * max depth
> * web site authentication
> * language
> * country
> * tags
> * collections
> * ...
>
> The pileline includes various ready to use stages (text
> extraction, language detection, Solr ready to index xml writer, ...).
>
> All is very configurable and extendible either by scripting or
> java coding.
>
> With scripting technology, you can help the crawler to handle
> javascript links or help the pipeline to extract relevant title
> and cleanup the html pages (remove menus, header, footers, ..)
>
> With java coding, you can develop your own pipeline stage stage
>
> The Crawl Anywhere web site provides good explanations and screen
> shots. All is documented in a wiki.
>
> The current version is 1.1.4. You can download and try it out from
> here : www.crawl-anywhere.com <http://www.crawl-anywhere.com>
>
>
> Regards
>
> Dominique
>
>
Re: [ANNOUNCE] Web Crawler
Posted by Lukáš Vlček <lu...@gmail.com>.
Hi,
is there any plan to open source it?
Regards,
Lukas
[OT] I tried HuriSearch, input "Java" into search field, it returned a lot
of references to coldfusion error pages. May be a recrawl would help?
On Wed, Mar 2, 2011 at 1:25 AM, Dominique Bejean
<do...@eolya.fr>wrote:
> Hi,
>
> I would like to announce Crawl Anywhere. Crawl-Anywhere is a Java Web
> Crawler. It includes :
>
> * a crawler
> * a document processing pipeline
> * a solr indexer
>
> The crawler has a web administration in order to manage web sites to be
> crawled. Each web site crawl is configured with a lot of possible parameters
> (no all mandatory) :
>
> * number of simultaneous items crawled by site
> * recrawl period rules based on item type (html, PDF, …)
> * item type inclusion / exclusion rules
> * item path inclusion / exclusion / strategy rules
> * max depth
> * web site authentication
> * language
> * country
> * tags
> * collections
> * ...
>
> The pileline includes various ready to use stages (text extraction,
> language detection, Solr ready to index xml writer, ...).
>
> All is very configurable and extendible either by scripting or java coding.
>
> With scripting technology, you can help the crawler to handle javascript
> links or help the pipeline to extract relevant title and cleanup the html
> pages (remove menus, header, footers, ..)
>
> With java coding, you can develop your own pipeline stage stage
>
> The Crawl Anywhere web site provides good explanations and screen shots.
> All is documented in a wiki.
>
> The current version is 1.1.4. You can download and try it out from here :
> www.crawl-anywhere.com
>
>
> Regards
>
> Dominique
>
>
Re: Crawl Anywhere -
Posted by "Markus.Mirsberger" <ma...@gmx.de>.
Hi,
did you try Heritrix?
The documents are stored as html inside an warc file which can be
postprocessed easily.
Cheers,
Markus
On 11.02.2013 12:16, SivaKarthik wrote:
> Dear Erick,
> Thanks for ur relpy..
> ya..nutch can meet my requirement...
> but the problem is, i want to store the crawled document in html or xml
> format instead of mapreduce format..
> not sure nutch plugins available to convert into xml files.
> please share me if you any idea .
>
> ThankYou
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p4039619.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Crawl Anywhere -
Posted by SUJIT PAL <su...@comcast.net>.
Hi Siva,
You will probably get a better reply if you head over to the nutch mailing list [http://nutch.apache.org/mailing_lists.html] and ask there.
Nutch 2.1 may be what you are looking for (stores pages in NoSQL database).
Regards,
Sujit
On Feb 10, 2013, at 9:16 PM, SivaKarthik wrote:
> Dear Erick,
> Thanks for ur relpy..
> ya..nutch can meet my requirement...
> but the problem is, i want to store the crawled document in html or xml
> format instead of mapreduce format..
> not sure nutch plugins available to convert into xml files.
> please share me if you any idea .
>
> ThankYou
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p4039619.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Crawl Anywhere -
Posted by Jan Høydahl <ja...@cominvent.com>.
Have a look at Nutch2, it is decoupled from HDFS and can store docs in e.g. HBase or other NoSql store.
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Solr Training - www.solrtraining.com
11. feb. 2013 kl. 06:16 skrev SivaKarthik <si...@gmail.com>:
> Dear Erick,
> Thanks for ur relpy..
> ya..nutch can meet my requirement...
> but the problem is, i want to store the crawled document in html or xml
> format instead of mapreduce format..
> not sure nutch plugins available to convert into xml files.
> please share me if you any idea .
>
> ThankYou
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p4039619.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Crawl Anywhere -
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
Crawl-Anywhere includes a customizable document processing pipeline.
Crawl-Anywhere can also cache original crawled pages and documents in a
mongodb database.
Best regards.
Dominique
Le 11/02/13 06:16, SivaKarthik a écrit :
> Dear Erick,
> Thanks for ur relpy..
> ya..nutch can meet my requirement...
> but the problem is, i want to store the crawled document in html or xml
> format instead of mapreduce format..
> not sure nutch plugins available to convert into xml files.
> please share me if you any idea .
>
> ThankYou
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p4039619.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Dominique Béjean
www.crawl-anywhere.com
Re: Crawl Anywhere -
Posted by SivaKarthik <si...@gmail.com>.
Dear Erick,
Thanks for ur relpy..
ya..nutch can meet my requirement...
but the problem is, i want to store the crawled document in html or xml
format instead of mapreduce format..
not sure nutch plugins available to convert into xml files.
please share me if you any idea .
ThankYou
--
View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p4039619.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Crawl Anywhere -
Posted by Erick Erickson <er...@gmail.com>.
Have you looked at nutch?
On Fri, Feb 8, 2013 at 12:25 AM, SivaKarthik
<si...@gmail.com>wrote:
> Hi All,
> in our project, we need to download around millions of pages...
> so is there any support to do the crawling in distributed environment
> using
> crawl-anywhere apps?
> or wat could be the alternatives...?
>
> Thanks in advance..
>
>
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p4039172.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Crawl Anywhere -
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
I didn't see this question.
Yes, I confirm Crawl-Anywhere can crawl in distributed environment.
If you have several huge web sites to crawl, you can dispatch crawling
across several crawler engines. However, one single web site can only be
crawled by one crawler engine at a time.
This limitation should be removed in future version.
For your information, new version 4.0.0 is now available as an
open-source project hosted on Github -
https://github.com/bejean/crawl-anywhere
Regards.
Le 11/02/13 12:02, O. Klein a écrit :
> Yes you can run CA on different machines.
>
> In "Manage" you have to set target and engine for this to work.
>
> I've never done this, so you have to contact the developer for more details.
>
>
>
> SivaKarthik wrote
>> Hi All,
>> in our project, we need to download around millions of pages...
>> so is there any support to do the crawling in distributed environment
>> using crawl-anywhere apps?
>> or wat could be the alternatives...?
>>
>> Thanks in advance..
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p4039674.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Dominique Béjean
+33 6 08 46 12 43
skype: dbejean
www.eolya.fr
www.crawl-anywhere.com
www.mysolrserver.com
Re: Crawl Anywhere -
Posted by "O. Klein" <kl...@octoweb.nl>.
Yes you can run CA on different machines.
In "Manage" you have to set target and engine for this to work.
I've never done this, so you have to contact the developer for more details.
SivaKarthik wrote
> Hi All,
> in our project, we need to download around millions of pages...
> so is there any support to do the crawling in distributed environment
> using crawl-anywhere apps?
> or wat could be the alternatives...?
>
> Thanks in advance..
--
View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p4039674.html
Sent from the Solr - User mailing list archive at Nabble.com.
Crawl Anywhere -
Posted by SivaKarthik <si...@gmail.com>.
Hi All,
in our project, we need to download around millions of pages...
so is there any support to do the crawling in distributed environment using
crawl-anywhere apps?
or wat could be the alternatives...?
Thanks in advance..
--
View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p4039172.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: [ANNOUNCE] Web Crawler
Posted by Dominique Bejean <do...@eolya.fr>.
David,
The UI was not the only reason that make me choose to write a totaly new
crawler. After eliminating candidate crawlers due to various reasons
(inactive project, ...), Nutch and Heritrix where the 2 crawlers in my
short list of possible candidates to be use.
In my mind, the crawler and the pipleline have to be tottaly
disconnected of the target repository (Solr, ...). This made nutch not a
possible choice.
At the end, I found Heritrix to far of the solution's architecture I
imagined.
Dominique
Le 02/03/11 05:41, David Smiley (@MITRE.org) a écrit :
> Dominique,
> The obvious number one question is of course why you re-invented this wheel
> when there are several existing crawlers to choose from. Your website says
> the reason is that the UIs on existing crawlers (e.g. Nutch, Heritrix, ...)
> weren't sufficiently user-friendly or had the site-specific configuration
> you wanted. Well if that is the case, why didn't you add/enhance such
> capabilities for an existing crawler?
>
> ~ David Smiley
>
> -----
> Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
Re: [ANNOUNCE] Web Crawler
Posted by "David Smiley (@MITRE.org)" <DS...@mitre.org>.
Dominique,
The obvious number one question is of course why you re-invented this wheel
when there are several existing crawlers to choose from. Your website says
the reason is that the UIs on existing crawlers (e.g. Nutch, Heritrix, ...)
weren't sufficiently user-friendly or had the site-specific configuration
you wanted. Well if that is the case, why didn't you add/enhance such
capabilities for an existing crawler?
~ David Smiley
-----
Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
--
View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607831p2608956.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: [ANNOUNCE] Web Crawler
Posted by Dominique Bejean <do...@eolya.fr>.
Aditya,
The crawler is not open source and won't be in the next future. Anyway,
I have to change the license because it can be use for any personal or
commercial projects.
Sincerely,
Dominique
Le 02/03/11 10:02, findbestopensource a écrit :
> Hello Dominique Bejean,
>
> Good job.
>
> We identified almost 8 open source web crawlers
> http://www.findbestopensource.com/tagged/webcrawler I don't know how
> far yours would be different from the rest.
>
> Your license states that it is not open source but it is free for
> personnel use.
>
> Regards
> Aditya
> www.findbestopensource.com <http://www.findbestopensource.com>
>
>
> On Wed, Mar 2, 2011 at 5:55 AM, Dominique Bejean
> <dominique.bejean@eolya.fr <ma...@eolya.fr>> wrote:
>
> Hi,
>
> I would like to announce Crawl Anywhere. Crawl-Anywhere is a Java
> Web Crawler. It includes :
>
> * a crawler
> * a document processing pipeline
> * a solr indexer
>
> The crawler has a web administration in order to manage web sites
> to be crawled. Each web site crawl is configured with a lot of
> possible parameters (no all mandatory) :
>
> * number of simultaneous items crawled by site
> * recrawl period rules based on item type (html, PDF, …)
> * item type inclusion / exclusion rules
> * item path inclusion / exclusion / strategy rules
> * max depth
> * web site authentication
> * language
> * country
> * tags
> * collections
> * ...
>
> The pileline includes various ready to use stages (text
> extraction, language detection, Solr ready to index xml writer, ...).
>
> All is very configurable and extendible either by scripting or
> java coding.
>
> With scripting technology, you can help the crawler to handle
> javascript links or help the pipeline to extract relevant title
> and cleanup the html pages (remove menus, header, footers, ..)
>
> With java coding, you can develop your own pipeline stage stage
>
> The Crawl Anywhere web site provides good explanations and screen
> shots. All is documented in a wiki.
>
> The current version is 1.1.4. You can download and try it out from
> here : www.crawl-anywhere.com <http://www.crawl-anywhere.com>
>
>
> Regards
>
> Dominique
>
>
Re: [ANNOUNCE] Web Crawler
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
Release 3.0.3 was tested with :
* Oracle Java 6 but should work fine with version 7
* Tomcat 5.5 and 6 and 7
* PHP 5.2.x and 5.3.x
* Apache 2.2.x
* MongoDB 64 bits 2.2 (know issue with 2.4)
The new release 4.0.0-alpha-2 is available under Github -
https://github.com/bejean/crawl-anywhere
The pre-requisites are :
Oracle Java 6 or >
Tomcat 5.5 or >
Apache 2.2 or >
PHP 5.2.x or 5.3.x or 5.4.x
MongoDB 64 bits 2.2 or >
Solr 3.x or > (configuration files provided for Solr 4.3.0)
And the up to date installation instructions are here
http://www.crawl-anywhere.com/installation-v400/
Please read the Github project home page, all information are provided.
Regards.
Dominique
Le 23/05/13 07:38, Rajesh Nikam a écrit :
> Hi,
>
> crawl anywhere seems to using old versions of java, tomcat, etc.
>
> http://www.crawl-anywhere.com/installation-v300/
>
> Will it work with new versions of these required software ?
>
> Is there updated installation guide available ?
>
> Thanks
> Rajesh
>
>
>
>
>
> On Wed, May 22, 2013 at 6:48 PM, Dominique Bejean
> <dominique.bejean@eolya.fr <ma...@eolya.fr>> wrote:
>
> Hi,
>
> Crawl-Anywhere is now open-source -
> https://github.com/bejean/crawl-anywhere
>
> Best regards.
>
>
> Le 02/03/11 10:02, findbestopensource a écrit :
>
> Hello Dominique Bejean,
>
> Good job.
>
> We identified almost 8 open source web crawlers
> http://www.findbestopensource.com/tagged/webcrawler I don't
> know how far yours would be different from the rest.
>
> Your license states that it is not open source but it is free
> for personnel use.
>
> Regards
> Aditya
> www.findbestopensource.com <http://www.findbestopensource.com>
> <http://www.findbestopensource.com>
>
>
> On Wed, Mar 2, 2011 at 5:55 AM, Dominique Bejean
> <dominique.bejean@eolya.fr <ma...@eolya.fr>
> <mailto:dominique.bejean@eolya.fr
> <ma...@eolya.fr>>> wrote:
>
> Hi,
>
> I would like to announce Crawl Anywhere. Crawl-Anywhere is
> a Java
> Web Crawler. It includes :
>
> * a crawler
> * a document processing pipeline
> * a solr indexer
>
> The crawler has a web administration in order to manage
> web sites
> to be crawled. Each web site crawl is configured with a lot of
> possible parameters (no all mandatory) :
>
> * number of simultaneous items crawled by site
> * recrawl period rules based on item type (html, PDF, …)
> * item type inclusion / exclusion rules
> * item path inclusion / exclusion / strategy rules
> * max depth
> * web site authentication
> * language
> * country
> * tags
> * collections
> * ...
>
> The pileline includes various ready to use stages (text
> extraction, language detection, Solr ready to index xml
> writer, ...).
>
> All is very configurable and extendible either by scripting or
> java coding.
>
> With scripting technology, you can help the crawler to handle
> javascript links or help the pipeline to extract relevant
> title
> and cleanup the html pages (remove menus, header, footers, ..)
>
> With java coding, you can develop your own pipeline stage
> stage
>
> The Crawl Anywhere web site provides good explanations and
> screen
> shots. All is documented in a wiki.
>
> The current version is 1.1.4. You can download and try it
> out from
> here : www.crawl-anywhere.com
> <http://www.crawl-anywhere.com> <http://www.crawl-anywhere.com>
>
>
> Regards
>
> Dominique
>
>
>
> --
> Dominique Béjean
> +33 6 08 46 12 43
> skype: dbejean
> www.eolya.fr <http://www.eolya.fr>
> www.crawl-anywhere.com <http://www.crawl-anywhere.com>
> www.mysolrserver.com <http://www.mysolrserver.com>
>
>
--
Dominique Béjean
+33 6 08 46 12 43
skype: dbejean
www.eolya.fr
www.crawl-anywhere.com
Re: [ANNOUNCE] Web Crawler
Posted by Rajesh Nikam <ra...@gmail.com>.
Hi,
crawl anywhere seems to using old versions of java, tomcat, etc.
http://www.crawl-anywhere.com/installation-v300/
Will it work with new versions of these required software ?
Is there updated installation guide available ?
Thanks
Rajesh
On Wed, May 22, 2013 at 6:48 PM, Dominique Bejean <dominique.bejean@eolya.fr
> wrote:
> Hi,
>
> Crawl-Anywhere is now open-source - https://github.com/bejean/**
> crawl-anywhere <https://github.com/bejean/crawl-anywhere>
>
> Best regards.
>
>
> Le 02/03/11 10:02, findbestopensource a écrit :
>
>> Hello Dominique Bejean,
>>
>> Good job.
>>
>> We identified almost 8 open source web crawlers
>> http://www.findbestopensource.**com/tagged/webcrawler<http://www.findbestopensource.com/tagged/webcrawler> I don't know how far yours would be different from the rest.
>>
>> Your license states that it is not open source but it is free for
>> personnel use.
>>
>> Regards
>> Aditya
>> www.findbestopensource.com <http://www.**findbestopensource.com<http://www.findbestopensource.com>
>> >
>>
>>
>> On Wed, Mar 2, 2011 at 5:55 AM, Dominique Bejean <
>> dominique.bejean@eolya.fr <ma...@eolya.fr>>>
>> wrote:
>>
>> Hi,
>>
>> I would like to announce Crawl Anywhere. Crawl-Anywhere is a Java
>> Web Crawler. It includes :
>>
>> * a crawler
>> * a document processing pipeline
>> * a solr indexer
>>
>> The crawler has a web administration in order to manage web sites
>> to be crawled. Each web site crawl is configured with a lot of
>> possible parameters (no all mandatory) :
>>
>> * number of simultaneous items crawled by site
>> * recrawl period rules based on item type (html, PDF, …)
>> * item type inclusion / exclusion rules
>> * item path inclusion / exclusion / strategy rules
>> * max depth
>> * web site authentication
>> * language
>> * country
>> * tags
>> * collections
>> * ...
>>
>> The pileline includes various ready to use stages (text
>> extraction, language detection, Solr ready to index xml writer, ...).
>>
>> All is very configurable and extendible either by scripting or
>> java coding.
>>
>> With scripting technology, you can help the crawler to handle
>> javascript links or help the pipeline to extract relevant title
>> and cleanup the html pages (remove menus, header, footers, ..)
>>
>> With java coding, you can develop your own pipeline stage stage
>>
>> The Crawl Anywhere web site provides good explanations and screen
>> shots. All is documented in a wiki.
>>
>> The current version is 1.1.4. You can download and try it out from
>> here : www.crawl-anywhere.com <http://www.crawl-anywhere.com**>
>>
>>
>> Regards
>>
>> Dominique
>>
>>
>>
> --
> Dominique Béjean
> +33 6 08 46 12 43
> skype: dbejean
> www.eolya.fr
> www.crawl-anywhere.com
> www.mysolrserver.com
>
>
Re: [ANNOUNCE] Web Crawler
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
Crawl-Anywhere is now open-source - https://github.com/bejean/crawl-anywhere
Best regards.
Le 02/03/11 10:02, findbestopensource a écrit :
> Hello Dominique Bejean,
>
> Good job.
>
> We identified almost 8 open source web crawlers
> http://www.findbestopensource.com/tagged/webcrawler I don't know how
> far yours would be different from the rest.
>
> Your license states that it is not open source but it is free for
> personnel use.
>
> Regards
> Aditya
> www.findbestopensource.com <http://www.findbestopensource.com>
>
>
> On Wed, Mar 2, 2011 at 5:55 AM, Dominique Bejean
> <dominique.bejean@eolya.fr <ma...@eolya.fr>> wrote:
>
> Hi,
>
> I would like to announce Crawl Anywhere. Crawl-Anywhere is a Java
> Web Crawler. It includes :
>
> * a crawler
> * a document processing pipeline
> * a solr indexer
>
> The crawler has a web administration in order to manage web sites
> to be crawled. Each web site crawl is configured with a lot of
> possible parameters (no all mandatory) :
>
> * number of simultaneous items crawled by site
> * recrawl period rules based on item type (html, PDF, …)
> * item type inclusion / exclusion rules
> * item path inclusion / exclusion / strategy rules
> * max depth
> * web site authentication
> * language
> * country
> * tags
> * collections
> * ...
>
> The pileline includes various ready to use stages (text
> extraction, language detection, Solr ready to index xml writer, ...).
>
> All is very configurable and extendible either by scripting or
> java coding.
>
> With scripting technology, you can help the crawler to handle
> javascript links or help the pipeline to extract relevant title
> and cleanup the html pages (remove menus, header, footers, ..)
>
> With java coding, you can develop your own pipeline stage stage
>
> The Crawl Anywhere web site provides good explanations and screen
> shots. All is documented in a wiki.
>
> The current version is 1.1.4. You can download and try it out from
> here : www.crawl-anywhere.com <http://www.crawl-anywhere.com>
>
>
> Regards
>
> Dominique
>
>
--
Dominique Béjean
+33 6 08 46 12 43
skype: dbejean
www.eolya.fr
www.crawl-anywhere.com
www.mysolrserver.com
Re: [ANNOUNCE] Web Crawler
Posted by Nestor Oviedo <ov...@gmail.com>.
Hi everyone!
I've been following this thread and I realized we've constructed something
similar to "Crawl Anywhere". The main difference is that our project is
oriented to the digital libraries and digital repositories context.
Specifically related to metadata collection from multiple sources,
information improvements and storing in multiple destinations.
So far, I can share an article about the project, because the code is in our
development machines and testing servers. If everything goes well, we plan
to make it open source in the near future.
I'd be glad to hear your comments and opinions about it. There is no need to
be polite.
Thanks in advance.
Best regards.
Nestor
On Wed, Mar 2, 2011 at 11:46 AM, Dominique Bejean <dominique.bejean@eolya.fr
> wrote:
> Hi,
>
> No, it doesn't. It looks like to be a apache httpclient 3.x limitation.
> https://issues.apache.org/jira/browse/HTTPCLIENT-579
>
> Dominique
>
> Le 02/03/11 15:04, Thumuluri, Sai a écrit :
>
> Dominique, Does your crawler support NTLM2 authentication? We have content
>> under SiteMinder which uses NTLM2 and that is posing challenges with Nutch?
>>
>> -----Original Message-----
>> From: Dominique Bejean [mailto:dominique.bejean@eolya.fr]
>> Sent: Wednesday, March 02, 2011 6:22 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: [ANNOUNCE] Web Crawler
>>
>> Aditya,
>>
>> The crawler is not open source and won't be in the next future. Anyway,
>> I have to change the license because it can be use for any personal or
>> commercial projects.
>>
>> Sincerely,
>>
>> Dominique
>>
>> Le 02/03/11 10:02, findbestopensource a écrit :
>>
>>> Hello Dominique Bejean,
>>>
>>> Good job.
>>>
>>> We identified almost 8 open source web crawlers
>>> http://www.findbestopensource.com/tagged/webcrawler I don't know how
>>> far yours would be different from the rest.
>>>
>>> Your license states that it is not open source but it is free for
>>> personnel use.
>>>
>>> Regards
>>> Aditya
>>> www.findbestopensource.com<http://www.findbestopensource.com>
>>>
>>>
>>> On Wed, Mar 2, 2011 at 5:55 AM, Dominique Bejean
>>> <do...@eolya.fr>> wrote:
>>>
>>> Hi,
>>>
>>> I would like to announce Crawl Anywhere. Crawl-Anywhere is a Java
>>> Web Crawler. It includes :
>>>
>>> * a crawler
>>> * a document processing pipeline
>>> * a solr indexer
>>>
>>> The crawler has a web administration in order to manage web sites
>>> to be crawled. Each web site crawl is configured with a lot of
>>> possible parameters (no all mandatory) :
>>>
>>> * number of simultaneous items crawled by site
>>> * recrawl period rules based on item type (html, PDF, ...)
>>> * item type inclusion / exclusion rules
>>> * item path inclusion / exclusion / strategy rules
>>> * max depth
>>> * web site authentication
>>> * language
>>> * country
>>> * tags
>>> * collections
>>> * ...
>>>
>>> The pileline includes various ready to use stages (text
>>> extraction, language detection, Solr ready to index xml writer, ...).
>>>
>>> All is very configurable and extendible either by scripting or
>>> java coding.
>>>
>>> With scripting technology, you can help the crawler to handle
>>> javascript links or help the pipeline to extract relevant title
>>> and cleanup the html pages (remove menus, header, footers, ..)
>>>
>>> With java coding, you can develop your own pipeline stage stage
>>>
>>> The Crawl Anywhere web site provides good explanations and screen
>>> shots. All is documented in a wiki.
>>>
>>> The current version is 1.1.4. You can download and try it out from
>>> here : www.crawl-anywhere.com<http://www.crawl-anywhere.com>
>>>
>>>
>>> Regards
>>>
>>> Dominique
>>>
>>>
>>>
Re: [ANNOUNCE] Web Crawler
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
No, it doesn't. It looks like to be a apache httpclient 3.x limitation.
https://issues.apache.org/jira/browse/HTTPCLIENT-579
Dominique
Le 02/03/11 15:04, Thumuluri, Sai a écrit :
> Dominique, Does your crawler support NTLM2 authentication? We have content under SiteMinder which uses NTLM2 and that is posing challenges with Nutch?
>
> -----Original Message-----
> From: Dominique Bejean [mailto:dominique.bejean@eolya.fr]
> Sent: Wednesday, March 02, 2011 6:22 AM
> To: solr-user@lucene.apache.org
> Subject: Re: [ANNOUNCE] Web Crawler
>
> Aditya,
>
> The crawler is not open source and won't be in the next future. Anyway,
> I have to change the license because it can be use for any personal or
> commercial projects.
>
> Sincerely,
>
> Dominique
>
> Le 02/03/11 10:02, findbestopensource a écrit :
>> Hello Dominique Bejean,
>>
>> Good job.
>>
>> We identified almost 8 open source web crawlers
>> http://www.findbestopensource.com/tagged/webcrawler I don't know how
>> far yours would be different from the rest.
>>
>> Your license states that it is not open source but it is free for
>> personnel use.
>>
>> Regards
>> Aditya
>> www.findbestopensource.com<http://www.findbestopensource.com>
>>
>>
>> On Wed, Mar 2, 2011 at 5:55 AM, Dominique Bejean
>> <do...@eolya.fr>> wrote:
>>
>> Hi,
>>
>> I would like to announce Crawl Anywhere. Crawl-Anywhere is a Java
>> Web Crawler. It includes :
>>
>> * a crawler
>> * a document processing pipeline
>> * a solr indexer
>>
>> The crawler has a web administration in order to manage web sites
>> to be crawled. Each web site crawl is configured with a lot of
>> possible parameters (no all mandatory) :
>>
>> * number of simultaneous items crawled by site
>> * recrawl period rules based on item type (html, PDF, ...)
>> * item type inclusion / exclusion rules
>> * item path inclusion / exclusion / strategy rules
>> * max depth
>> * web site authentication
>> * language
>> * country
>> * tags
>> * collections
>> * ...
>>
>> The pileline includes various ready to use stages (text
>> extraction, language detection, Solr ready to index xml writer, ...).
>>
>> All is very configurable and extendible either by scripting or
>> java coding.
>>
>> With scripting technology, you can help the crawler to handle
>> javascript links or help the pipeline to extract relevant title
>> and cleanup the html pages (remove menus, header, footers, ..)
>>
>> With java coding, you can develop your own pipeline stage stage
>>
>> The Crawl Anywhere web site provides good explanations and screen
>> shots. All is documented in a wiki.
>>
>> The current version is 1.1.4. You can download and try it out from
>> here : www.crawl-anywhere.com<http://www.crawl-anywhere.com>
>>
>>
>> Regards
>>
>> Dominique
>>
>>
RE: [ANNOUNCE] Web Crawler
Posted by "Thumuluri, Sai" <Sa...@VerizonWireless.com>.
Dominique, Does your crawler support NTLM2 authentication? We have content under SiteMinder which uses NTLM2 and that is posing challenges with Nutch?
-----Original Message-----
From: Dominique Bejean [mailto:dominique.bejean@eolya.fr]
Sent: Wednesday, March 02, 2011 6:22 AM
To: solr-user@lucene.apache.org
Subject: Re: [ANNOUNCE] Web Crawler
Aditya,
The crawler is not open source and won't be in the next future. Anyway,
I have to change the license because it can be use for any personal or
commercial projects.
Sincerely,
Dominique
Le 02/03/11 10:02, findbestopensource a écrit :
> Hello Dominique Bejean,
>
> Good job.
>
> We identified almost 8 open source web crawlers
> http://www.findbestopensource.com/tagged/webcrawler I don't know how
> far yours would be different from the rest.
>
> Your license states that it is not open source but it is free for
> personnel use.
>
> Regards
> Aditya
> www.findbestopensource.com <http://www.findbestopensource.com>
>
>
> On Wed, Mar 2, 2011 at 5:55 AM, Dominique Bejean
> <dominique.bejean@eolya.fr <ma...@eolya.fr>> wrote:
>
> Hi,
>
> I would like to announce Crawl Anywhere. Crawl-Anywhere is a Java
> Web Crawler. It includes :
>
> * a crawler
> * a document processing pipeline
> * a solr indexer
>
> The crawler has a web administration in order to manage web sites
> to be crawled. Each web site crawl is configured with a lot of
> possible parameters (no all mandatory) :
>
> * number of simultaneous items crawled by site
> * recrawl period rules based on item type (html, PDF, ...)
> * item type inclusion / exclusion rules
> * item path inclusion / exclusion / strategy rules
> * max depth
> * web site authentication
> * language
> * country
> * tags
> * collections
> * ...
>
> The pileline includes various ready to use stages (text
> extraction, language detection, Solr ready to index xml writer, ...).
>
> All is very configurable and extendible either by scripting or
> java coding.
>
> With scripting technology, you can help the crawler to handle
> javascript links or help the pipeline to extract relevant title
> and cleanup the html pages (remove menus, header, footers, ..)
>
> With java coding, you can develop your own pipeline stage stage
>
> The Crawl Anywhere web site provides good explanations and screen
> shots. All is documented in a wiki.
>
> The current version is 1.1.4. You can download and try it out from
> here : www.crawl-anywhere.com <http://www.crawl-anywhere.com>
>
>
> Regards
>
> Dominique
>
>
Re: [ANNOUNCE] Web Crawler
Posted by findbestopensource <fi...@gmail.com>.
Hello Dominique Bejean,
Good job.
We identified almost 8 open source web crawlers
http://www.findbestopensource.com/tagged/webcrawler I don't know how far
yours would be different from the rest.
Your license states that it is not open source but it is free for personnel
use.
Regards
Aditya
www.findbestopensource.com
On Wed, Mar 2, 2011 at 5:55 AM, Dominique Bejean
<do...@eolya.fr>wrote:
> Hi,
>
> I would like to announce Crawl Anywhere. Crawl-Anywhere is a Java Web
> Crawler. It includes :
>
> * a crawler
> * a document processing pipeline
> * a solr indexer
>
> The crawler has a web administration in order to manage web sites to be
> crawled. Each web site crawl is configured with a lot of possible parameters
> (no all mandatory) :
>
> * number of simultaneous items crawled by site
> * recrawl period rules based on item type (html, PDF, …)
> * item type inclusion / exclusion rules
> * item path inclusion / exclusion / strategy rules
> * max depth
> * web site authentication
> * language
> * country
> * tags
> * collections
> * ...
>
> The pileline includes various ready to use stages (text extraction,
> language detection, Solr ready to index xml writer, ...).
>
> All is very configurable and extendible either by scripting or java coding.
>
> With scripting technology, you can help the crawler to handle javascript
> links or help the pipeline to extract relevant title and cleanup the html
> pages (remove menus, header, footers, ..)
>
> With java coding, you can develop your own pipeline stage stage
>
> The Crawl Anywhere web site provides good explanations and screen shots.
> All is documented in a wiki.
>
> The current version is 1.1.4. You can download and try it out from here :
> www.crawl-anywhere.com
>
>
> Regards
>
> Dominique
>
>
Re: [ANNOUNCE] Web Crawler
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
The crawler comes with a extendible document processing pipeline. If you
know java libraries or web services for 'wrapper induction' processing,
it is possible to implement a dedicated stage in the pipeline.
Dominique
Le 02/03/11 12:20, Geert-Jan Brits a écrit :
> Hi Dominique,
>
> This looks nice.
> In the past, I've been interested in (semi)-automatically inducing a
> scheme/wrapper from a set of example webpages (often called 'wrapper
> induction' is the scientific field) .
> This would allow for fast scheme-creation which could be used as a
> basis for extraction.
>
> Lately I've been looking for crawlers that incoporate this technology
> but without success.
> Any plans on incorporating this?
>
> Cheers,
> Geert-Jan
>
> 2011/3/2 Dominique Bejean <dominique.bejean@eolya.fr
> <ma...@eolya.fr>>
>
> Rosa,
>
> In the pipeline, there is a stage that extract the text from the
> original document (PDF, HTML, ...).
> It is possible to plug scripts (Java 6 compliant) in order to keep
> only relevant parts of the document.
> See
> http://www.wiizio.com/confluence/display/CRAWLUSERS/DocTextExtractor+stage
>
> Dominique
>
> Le 02/03/11 09:36, Rosa (Anuncios) a écrit :
>
> Nice job!
>
> It would be good to be able to extract specific data from a
> given page via XPATH though.
>
> Regards,
>
>
> Le 02/03/2011 01:25, Dominique Bejean a écrit :
>
> Hi,
>
> I would like to announce Crawl Anywhere. Crawl-Anywhere is
> a Java Web Crawler. It includes :
>
> * a crawler
> * a document processing pipeline
> * a solr indexer
>
> The crawler has a web administration in order to manage
> web sites to be crawled. Each web site crawl is configured
> with a lot of possible parameters (no all mandatory) :
>
> * number of simultaneous items crawled by site
> * recrawl period rules based on item type (html, PDF, …)
> * item type inclusion / exclusion rules
> * item path inclusion / exclusion / strategy rules
> * max depth
> * web site authentication
> * language
> * country
> * tags
> * collections
> * ...
>
> The pileline includes various ready to use stages (text
> extraction, language detection, Solr ready to index xml
> writer, ...).
>
> All is very configurable and extendible either by
> scripting or java coding.
>
> With scripting technology, you can help the crawler to
> handle javascript links or help the pipeline to extract
> relevant title and cleanup the html pages (remove menus,
> header, footers, ..)
>
> With java coding, you can develop your own pipeline stage
> stage
>
> The Crawl Anywhere web site provides good explanations and
> screen shots. All is documented in a wiki.
>
> The current version is 1.1.4. You can download and try it
> out from here : www.crawl-anywhere.com
> <http://www.crawl-anywhere.com>
>
>
> Regards
>
> Dominique
>
>
>
>
>
Re: [ANNOUNCE] Web Crawler
Posted by Paul Libbrecht <pa...@hoplahup.net>.
VIewing the indexing result, which is a part of what you are describing I think, is a nice job for such an indexing framework.
Do you guys know whether such feature is already out there?
paul
Le 2 mars 2011 à 12:20, Geert-Jan Brits a écrit :
> Hi Dominique,
>
> This looks nice.
> In the past, I've been interested in (semi)-automatically inducing a
> scheme/wrapper from a set of example webpages (often called 'wrapper
> induction' is the scientific field) .
> This would allow for fast scheme-creation which could be used as a basis for
> extraction.
>
> Lately I've been looking for crawlers that incoporate this technology but
> without success.
> Any plans on incorporating this?
>
> Cheers,
> Geert-Jan
>
> 2011/3/2 Dominique Bejean <do...@eolya.fr>
>
>> Rosa,
>>
>> In the pipeline, there is a stage that extract the text from the original
>> document (PDF, HTML, ...).
>> It is possible to plug scripts (Java 6 compliant) in order to keep only
>> relevant parts of the document.
>> See
>> http://www.wiizio.com/confluence/display/CRAWLUSERS/DocTextExtractor+stage
>>
>> Dominique
>>
>> Le 02/03/11 09:36, Rosa (Anuncios) a écrit :
>>
>> Nice job!
>>>
>>> It would be good to be able to extract specific data from a given page via
>>> XPATH though.
>>>
>>> Regards,
>>>
>>>
>>> Le 02/03/2011 01:25, Dominique Bejean a écrit :
>>>
>>>> Hi,
>>>>
>>>> I would like to announce Crawl Anywhere. Crawl-Anywhere is a Java Web
>>>> Crawler. It includes :
>>>>
>>>> * a crawler
>>>> * a document processing pipeline
>>>> * a solr indexer
>>>>
>>>> The crawler has a web administration in order to manage web sites to be
>>>> crawled. Each web site crawl is configured with a lot of possible parameters
>>>> (no all mandatory) :
>>>>
>>>> * number of simultaneous items crawled by site
>>>> * recrawl period rules based on item type (html, PDF, …)
>>>> * item type inclusion / exclusion rules
>>>> * item path inclusion / exclusion / strategy rules
>>>> * max depth
>>>> * web site authentication
>>>> * language
>>>> * country
>>>> * tags
>>>> * collections
>>>> * ...
>>>>
>>>> The pileline includes various ready to use stages (text extraction,
>>>> language detection, Solr ready to index xml writer, ...).
>>>>
>>>> All is very configurable and extendible either by scripting or java
>>>> coding.
>>>>
>>>> With scripting technology, you can help the crawler to handle javascript
>>>> links or help the pipeline to extract relevant title and cleanup the html
>>>> pages (remove menus, header, footers, ..)
>>>>
>>>> With java coding, you can develop your own pipeline stage stage
>>>>
>>>> The Crawl Anywhere web site provides good explanations and screen shots.
>>>> All is documented in a wiki.
>>>>
>>>> The current version is 1.1.4. You can download and try it out from here :
>>>> www.crawl-anywhere.com
>>>>
>>>>
>>>> Regards
>>>>
>>>> Dominique
>>>>
>>>>
>>>>
>>>
>>>
Re: [ANNOUNCE] Web Crawler
Posted by Geert-Jan Brits <gb...@gmail.com>.
Hi Dominique,
This looks nice.
In the past, I've been interested in (semi)-automatically inducing a
scheme/wrapper from a set of example webpages (often called 'wrapper
induction' is the scientific field) .
This would allow for fast scheme-creation which could be used as a basis for
extraction.
Lately I've been looking for crawlers that incoporate this technology but
without success.
Any plans on incorporating this?
Cheers,
Geert-Jan
2011/3/2 Dominique Bejean <do...@eolya.fr>
> Rosa,
>
> In the pipeline, there is a stage that extract the text from the original
> document (PDF, HTML, ...).
> It is possible to plug scripts (Java 6 compliant) in order to keep only
> relevant parts of the document.
> See
> http://www.wiizio.com/confluence/display/CRAWLUSERS/DocTextExtractor+stage
>
> Dominique
>
> Le 02/03/11 09:36, Rosa (Anuncios) a écrit :
>
> Nice job!
>>
>> It would be good to be able to extract specific data from a given page via
>> XPATH though.
>>
>> Regards,
>>
>>
>> Le 02/03/2011 01:25, Dominique Bejean a écrit :
>>
>>> Hi,
>>>
>>> I would like to announce Crawl Anywhere. Crawl-Anywhere is a Java Web
>>> Crawler. It includes :
>>>
>>> * a crawler
>>> * a document processing pipeline
>>> * a solr indexer
>>>
>>> The crawler has a web administration in order to manage web sites to be
>>> crawled. Each web site crawl is configured with a lot of possible parameters
>>> (no all mandatory) :
>>>
>>> * number of simultaneous items crawled by site
>>> * recrawl period rules based on item type (html, PDF, …)
>>> * item type inclusion / exclusion rules
>>> * item path inclusion / exclusion / strategy rules
>>> * max depth
>>> * web site authentication
>>> * language
>>> * country
>>> * tags
>>> * collections
>>> * ...
>>>
>>> The pileline includes various ready to use stages (text extraction,
>>> language detection, Solr ready to index xml writer, ...).
>>>
>>> All is very configurable and extendible either by scripting or java
>>> coding.
>>>
>>> With scripting technology, you can help the crawler to handle javascript
>>> links or help the pipeline to extract relevant title and cleanup the html
>>> pages (remove menus, header, footers, ..)
>>>
>>> With java coding, you can develop your own pipeline stage stage
>>>
>>> The Crawl Anywhere web site provides good explanations and screen shots.
>>> All is documented in a wiki.
>>>
>>> The current version is 1.1.4. You can download and try it out from here :
>>> www.crawl-anywhere.com
>>>
>>>
>>> Regards
>>>
>>> Dominique
>>>
>>>
>>>
>>
>>
Re: [ANNOUNCE] Web Crawler
Posted by Dominique Bejean <do...@eolya.fr>.
Rosa,
In the pipeline, there is a stage that extract the text from the
original document (PDF, HTML, ...).
It is possible to plug scripts (Java 6 compliant) in order to keep only
relevant parts of the document.
See
http://www.wiizio.com/confluence/display/CRAWLUSERS/DocTextExtractor+stage
Dominique
Le 02/03/11 09:36, Rosa (Anuncios) a écrit :
> Nice job!
>
> It would be good to be able to extract specific data from a given page
> via XPATH though.
>
> Regards,
>
>
> Le 02/03/2011 01:25, Dominique Bejean a écrit :
>> Hi,
>>
>> I would like to announce Crawl Anywhere. Crawl-Anywhere is a Java Web
>> Crawler. It includes :
>>
>> * a crawler
>> * a document processing pipeline
>> * a solr indexer
>>
>> The crawler has a web administration in order to manage web sites to
>> be crawled. Each web site crawl is configured with a lot of possible
>> parameters (no all mandatory) :
>>
>> * number of simultaneous items crawled by site
>> * recrawl period rules based on item type (html, PDF, …)
>> * item type inclusion / exclusion rules
>> * item path inclusion / exclusion / strategy rules
>> * max depth
>> * web site authentication
>> * language
>> * country
>> * tags
>> * collections
>> * ...
>>
>> The pileline includes various ready to use stages (text extraction,
>> language detection, Solr ready to index xml writer, ...).
>>
>> All is very configurable and extendible either by scripting or java
>> coding.
>>
>> With scripting technology, you can help the crawler to handle
>> javascript links or help the pipeline to extract relevant title and
>> cleanup the html pages (remove menus, header, footers, ..)
>>
>> With java coding, you can develop your own pipeline stage stage
>>
>> The Crawl Anywhere web site provides good explanations and screen
>> shots. All is documented in a wiki.
>>
>> The current version is 1.1.4. You can download and try it out from
>> here : www.crawl-anywhere.com
>>
>>
>> Regards
>>
>> Dominique
>>
>>
>
>
Re: [ANNOUNCE] Web Crawler
Posted by "Rosa (Anuncios)" <ro...@gmail.com>.
Nice job!
It would be good to be able to extract specific data from a given page
via XPATH though.
Regards,
Le 02/03/2011 01:25, Dominique Bejean a écrit :
> Hi,
>
> I would like to announce Crawl Anywhere. Crawl-Anywhere is a Java Web
> Crawler. It includes :
>
> * a crawler
> * a document processing pipeline
> * a solr indexer
>
> The crawler has a web administration in order to manage web sites to
> be crawled. Each web site crawl is configured with a lot of possible
> parameters (no all mandatory) :
>
> * number of simultaneous items crawled by site
> * recrawl period rules based on item type (html, PDF, …)
> * item type inclusion / exclusion rules
> * item path inclusion / exclusion / strategy rules
> * max depth
> * web site authentication
> * language
> * country
> * tags
> * collections
> * ...
>
> The pileline includes various ready to use stages (text extraction,
> language detection, Solr ready to index xml writer, ...).
>
> All is very configurable and extendible either by scripting or java
> coding.
>
> With scripting technology, you can help the crawler to handle
> javascript links or help the pipeline to extract relevant title and
> cleanup the html pages (remove menus, header, footers, ..)
>
> With java coding, you can develop your own pipeline stage stage
>
> The Crawl Anywhere web site provides good explanations and screen
> shots. All is documented in a wiki.
>
> The current version is 1.1.4. You can download and try it out from
> here : www.crawl-anywhere.com
>
>
> Regards
>
> Dominique
>
>