Robot Redesign

[Fuad Efendi <fu...@efendi.ca>]

Hi All,

 

 

Ok, let’s push something above average into this aged topic,

What about Robot?

 

Recently I browsed Nutch (and Yahoo invested a lot in India…); and Bixo (and
I invested something…)

 

Both still have problems, and (if you are really trying to be “Polite
Robot”): 



1.       Don’t follow http://www.robotstxt.org – this is privately owned
website, information there is outdated at least 12 years, don’t forget to
click Google Ads.

2.       There are some new standards, and all such standards were pushed…
by robots!

3.       …. A LOT!!!

 

I was thinking about coding style… projects such as a NUtch, SOLR, BIXO,
Cascading… use static classes, so that codebase seems very small, single
class can do 10 times more that yours 100 classes…

 

But I think it’s better to improve existing codebase instead of complete
rewrite (and, very bad: copy-paste!)

 

Here are some improvements which I am going to work on, tell me if you are
interested:

 

-          Speel Check: user-agent, useragent, usreagent, …….

-          UTF8: according to specs, URL must be decoded before applying
rule from robots.txt; additionally, %2F need not be decoded! 

 

For instance, both Nutch and BIXO rely on Droids, but nothing happens…

 

I think framework should be clear enough so that we can add new rules (such
as “recrawl rate” or “sitemap” of even “custom domain-specific rules” (such
as Nutch RegEx Filter)) 

 

I want to push some code but I think it’s much better to follow Nutch coding
style (local/static/private) instead of this extremely naïve “interface &
implementation”…

 

Thanks,

 

 

Fuad Efendi

+1 416-993-2060

 <http://www.linkedin.com/in/liferay> http://www.linkedin.com/in/liferay

 

Tokenizer Inc.

 <http://www.tokenizer.ca/> http://www.tokenizer.ca/

Data Mining, Vertical Search

 

(sorry for Search Engine Optimization trick… but it is so popular here!)

 

Re: Robot Redesign

[florent andré <fl...@4sengines.com>]

Hi,

On 11/27/2010 04:01 PM, Fuad Efendi wrote:
> Ok; just no offence please, some people are doing it for glory, others
> simply paid fulltime by their employers (such as Yahoo, CNET, Oracle). I am
> cleaning fulltime some crap in a bank for instance. Sorry for off-topic.
> Robots are my part time hobby.
> 
> Real-life crawls (thousands websites) usually shows many specific directives
> which webmaster use for some robots, it's hard to find documentation
> (Japanese robots, Russian robots, Chinese, Americans :))
> 
> I think framework should be simple enough to plugin specific RobotDirective
> instance (or implementation?) - I don't see Nutch or BIXO approach can do it
> easily... also, Fetching Application (such as Nutch) can use additional
> directives from configuration files (such as do not follow anything like
> *shopping_cart* for specific site); it can be Regular Expressions, anything.

see this :
http://svn.apache.org/repos/asf/incubator/droids/trunk/droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java

Particulary this part :
URLFiltersFactory filtersFactory = new URLFiltersFactory();
    RegexURLFilter defaultURLFilter = new RegexURLFilter();
    defaultURLFilter.setFile("classpath:/regex-urlfilter.txt");

Fill your requirements no ?

> And I also love Wicket (off-topic again) but "scraper" (configurator) for a
> robot should have UI, and Wicket is fastest (development time)

Search in ML archive and/or bugtrack, there is someone (sorry don't
remember name), that build an ui configurator if I well remember...

++

> 
> 

RE: Robot Redesign

[Fuad Efendi <fu...@efendi.ca>]

Ok; just no offence please, some people are doing it for glory, others
simply paid fulltime by their employers (such as Yahoo, CNET, Oracle). I am
cleaning fulltime some crap in a bank for instance. Sorry for off-topic.
Robots are my part time hobby.

Real-life crawls (thousands websites) usually shows many specific directives
which webmaster use for some robots, it's hard to find documentation
(Japanese robots, Russian robots, Chinese, Americans :))

I think framework should be simple enough to plugin specific RobotDirective
instance (or implementation?) - I don't see Nutch or BIXO approach can do it
easily... also, Fetching Application (such as Nutch) can use additional
directives from configuration files (such as do not follow anything like
*shopping_cart* for specific site); it can be Regular Expressions, anything.
And I also love Wicket (off-topic again) but "scraper" (configurator) for a
robot should have UI, and Wicket is fastest (development time)