You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Sergey A Volkov <se...@gmail.com> on 2011/11/16 01:39:40 UTC
Nutch project and my Ph.D. thesis.
Hi!
I am postgraduate student in Saint Petersburg State University. I was
working with Nutch for about 3 years, have written my graduate work
based on it, and now I don't know what to do in my Ph.D work. (Nobody in
my department (System Programming) deals with web crawling)
I hope someone knows problems in web crawling, whose solutions can help
Nutch project and me in my future Ph.D. thesis.
Any ideas?
Thanks,
Sergey.
Re: Nutch project and my Ph.D. thesis.
Posted by Sergey A Volkov <se...@gmail.com>.
Thank you for your reply!
Looks like at first I should read this book.
I'll came back with my thought after this=)
Sergey.
On Wed 16 Nov 2011 04:11:54 PM MSK, Markus Jelsma wrote:
> Hi Sergey,
>
> The most profound problems or interesting stuff we've encountered are:
> - dealing with dynamic URL's such as calendars, also known as spider traps;
> - detecting duplicates based on sub-domain, many sites allow www, ww, wwww or
> everything else, you have to deal with it;
> - normalizing of URL's, highly important as it already prevents a lot of
> duplicates
> - various kinds of link analysis
> - detecting spam (link spam, content spam, various techniques)
> - general crawler ethics
> - dynamic politeness, large sites can be crawled more intense than small sites
> - deep and or shallow crawling, is coverage or freshness more important
>
> For me Bing Lui's excellent book on Web and Data mining gives a lot of
> insights. The best thing is that the author provides a royal list of
> references to highly interesting papers that you can then find online.
>
> In my opinion this book is mandatory when one is serious with web crawling.
>
> [1]: http://www.cs.uic.edu/~liub/WebMiningBook.html
>
> Good luck!
> Markus
>
> On Wednesday 16 November 2011 01:51:20 Sergey A Volkov wrote:
>> Hi!
>>
>> I am postgraduate student in Saint Petersburg State University. I was
>> working with Nutch for about 3 years, have written my graduate work
>> based on it, and now I don't know what to do in my Ph.D work. (Nobody in
>> my department (System Programming) deals with web crawling)
>>
>> I hope someone knows problems in web crawling, whose solutions can help
>> Nutch project and me in my future Ph.D. thesis. Any ideas?
>>
>> Thanks,
>> Sergey.
>
Re: Nutch project and my Ph.D. thesis.
Posted by Markus Jelsma <ma...@openindex.io>.
Hi Sergey,
The most profound problems or interesting stuff we've encountered are:
- dealing with dynamic URL's such as calendars, also known as spider traps;
- detecting duplicates based on sub-domain, many sites allow www, ww, wwww or
everything else, you have to deal with it;
- normalizing of URL's, highly important as it already prevents a lot of
duplicates
- various kinds of link analysis
- detecting spam (link spam, content spam, various techniques)
- general crawler ethics
- dynamic politeness, large sites can be crawled more intense than small sites
- deep and or shallow crawling, is coverage or freshness more important
For me Bing Lui's excellent book on Web and Data mining gives a lot of
insights. The best thing is that the author provides a royal list of
references to highly interesting papers that you can then find online.
In my opinion this book is mandatory when one is serious with web crawling.
[1]: http://www.cs.uic.edu/~liub/WebMiningBook.html
Good luck!
Markus
On Wednesday 16 November 2011 01:51:20 Sergey A Volkov wrote:
> Hi!
>
> I am postgraduate student in Saint Petersburg State University. I was
> working with Nutch for about 3 years, have written my graduate work
> based on it, and now I don't know what to do in my Ph.D work. (Nobody in
> my department (System Programming) deals with web crawling)
>
> I hope someone knows problems in web crawling, whose solutions can help
> Nutch project and me in my future Ph.D. thesis. Any ideas?
>
> Thanks,
> Sergey.
--
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350
Nutch project and my Ph.D. thesis.
Posted by Sergey A Volkov <se...@gmail.com>.
Hi!
I am postgraduate student in Saint Petersburg State University. I was
working with Nutch for about 3 years, have written my graduate work
based on it, and now I don't know what to do in my Ph.D work. (Nobody in
my department (System Programming) deals with web crawling)
I hope someone knows problems in web crawling, whose solutions can help
Nutch project and me in my future Ph.D. thesis. Any ideas?
Thanks,
Sergey.
Re: Fwd: Nutch project and my Ph.D. thesis.
Posted by Sergey A Volkov <se...@gmail.com>.
Hi Sebastian!
Glad to hear that you had read my work.
*Generating URL filters.*
This part of my work was not clearly described and out of date in my thesis.
Main idea is to reduce crawldb size (reducing junk in fetch list is side
effect).
Suppose we have some method to detect that downloaded document is "useful"
Then we consider that pages with enough outlinks (e.g. 10) to useful
documents are "semi-useful"
After that we can count number of "useful" and "semiuseful" pages for
each url prefix (each url splits to set of its prefixes), and add prefix
to filter using some heuristics (I had blocked prefixes without any
useful or semiuseful urls)
Actually it have not worked on production as good as I have supposed =(
Maybe because using discrete value for "usefullness" instead of float is
not a good idea.
Much more efficient in my case was adding robots.txt filters as url-filter.
*Deleting duplicates.
*This tweak works really good in very specific case, when I don't need
to update documents in index. (I'm working with news sites, so for me it
was ok).
Main idea is use key value storage instead of DeleteDuplicates. In this
case we don't have to read full index.
This is not MapReduce way, but it is simple and works really good,
because even with very limited resources kv storages can handle few
query per ms (we transfer only url+hash to server), so indexing
performance changes insignificantly.
There is no need in kv storage if lucene can store fields in separate
files, but as far as I know it can't.
>Did you work with TextProfileSignature or MD5Signature?
I have used md5 signature of cleaned text. It's not like in
TextProfileSignature but serves for similar purpose.
Thanks,
Sergey.
On 11/26/2011 01:36 AM, Sebastian Nagel wrote:
> Hi Sergey,
>
> a late answer, but I just read your work and found it very interesting
> and
> inspiring, especially your description of "a system for the automatic
> construction of URL filters". Why? - We recently had to setup for
> a customer URL filter and normalization rules to limit the number
> of crawled documents to a reasonable number. URLs were long (>300 chars)
> and contained many query parameters. Hard work and I thought that
> it would be a nice machine learning problem with
> - some similarity measure based on content data on the one side
> - URL features (path and query parts, host, etc.) on the other side
> and the question: Which parts of the URL lead to (near) duplicates and
> can or should be removed. Such, the target would be to find URL
> normalization rules, not filters.
> Your algorithm transferred to my problem would roughly mean:
> if addition or variation of one parameter does not lead
> to new interesting links the parameter could be skipped.
> I have to think about it. Unhappily, there is few time at work
> to solve nice machine learning problems.
>
> Just for all others: Sergey's algorithm eliminates URLs
> (by constructing URL filters) by counting the number of good links
> with this prefix.
> A question whether my understanding is right:
> I guess "u_p - number of "useful" links with the given prefix"
> means that there are u_p useful links pointing to URLs with this prefix.
> Or is it the opposite: outlinks pointing from documents with the prefix
> to useful documents.
>
> You also adressed duplicate detection performed on an earlier
> step (realized as indexing filter not operating on the index).
> I think this is of general interest. Just one further question:
> Did you work with TextProfileSignature or MD5Signature?
>
> And finally, as Markus pointed out there are many problems
> of common and/or academic interest around crawlers and links.
> I think the most important thing for an PhD is to find a problem
> without a bulk of papers being written about it.
>
> So, hope to hear more from you.
>
> Sebastian
>
> On 11/16/2011 03:06 AM, Sergey A Volkov wrote:
>> Thanks!
>>
>> Unfortunately my work is only in russian - anyway here is the link
>> https://github.com/volkov/diploma/blob/master/main.pdf?raw=true
>> Actually it is technical work contains some specific optimizations
>> for crawling news sites. At this
>> moment my english writing skills aren't good, but i'll try to
>> represent some interesting aspects of
>> my graduate paper somehow=)
>>
>> On Wed 16 Nov 2011 05:16:05 AM MSK, Lewis John Mcgibbney wrote:
>>> ---------- Forwarded message ----------
>>> From: Lewis John Mcgibbney<le...@gmail.com>
>>> Date: Wed, Nov 16, 2011 at 1:15 AM
>>> Subject: Re: Nutch project and my Ph.D. thesis.
>>> To: dev@nutch.apache.org
>>>
>>>
>>> Hi Sergey,
>>>
>>> There was a Professor from somewhere in S America that posted recently
>>> rearding some work he did, if you search the archives you may get a
>>> taster
>>> for work related to Nutch.
>>>
>>> Also can you provide a link to your work? I would be very intersted in
>>> having a look at the areas you have been working on. Also feel free
>>> to add
>>> your work to the wiki page references for others to see.
>>>
>>> Thank you.
>>>
>>> http://wiki.apache.org/nutch/AcademicArticles
>>>
>>>
>>> On Wed, Nov 16, 2011 at 12:39 AM, Sergey A
>>> Volkov<sergey.volkov.a@gmail.com
>>>> wrote:
>>>
>>>> Hi!
>>>>
>>>> I am postgraduate student in Saint Petersburg State University. I was
>>>> working with Nutch for about 3 years, have written my graduate work
>>>> based
>>>> on it, and now I don't know what to do in my Ph.D work. (Nobody in my
>>>> department (System Programming) deals with web crawling)
>>>>
>>>> I hope someone knows problems in web crawling, whose solutions can
>>>> help
>>>> Nutch project and me in my future Ph.D. thesis.
>>>>
>>>> Any ideas?
>>>>
>>>> Thanks,
>>>> Sergey.
>>>>
>>>
>>>
>>>
>>
>>
>
Re: Fwd: Nutch project and my Ph.D. thesis.
Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi Sergey,
a late answer, but I just read your work and found it very interesting and
inspiring, especially your description of "a system for the automatic
construction of URL filters". Why? - We recently had to setup for
a customer URL filter and normalization rules to limit the number
of crawled documents to a reasonable number. URLs were long (>300 chars)
and contained many query parameters. Hard work and I thought that
it would be a nice machine learning problem with
- some similarity measure based on content data on the one side
- URL features (path and query parts, host, etc.) on the other side
and the question: Which parts of the URL lead to (near) duplicates and
can or should be removed. Such, the target would be to find URL
normalization rules, not filters.
Your algorithm transferred to my problem would roughly mean:
if addition or variation of one parameter does not lead
to new interesting links the parameter could be skipped.
I have to think about it. Unhappily, there is few time at work
to solve nice machine learning problems.
Just for all others: Sergey's algorithm eliminates URLs
(by constructing URL filters) by counting the number of good links
with this prefix.
A question whether my understanding is right:
I guess "u_p - number of "useful" links with the given prefix"
means that there are u_p useful links pointing to URLs with this prefix.
Or is it the opposite: outlinks pointing from documents with the prefix
to useful documents.
You also adressed duplicate detection performed on an earlier
step (realized as indexing filter not operating on the index).
I think this is of general interest. Just one further question:
Did you work with TextProfileSignature or MD5Signature?
And finally, as Markus pointed out there are many problems
of common and/or academic interest around crawlers and links.
I think the most important thing for an PhD is to find a problem
without a bulk of papers being written about it.
So, hope to hear more from you.
Sebastian
On 11/16/2011 03:06 AM, Sergey A Volkov wrote:
> Thanks!
>
> Unfortunately my work is only in russian - anyway here is the link
> https://github.com/volkov/diploma/blob/master/main.pdf?raw=true
> Actually it is technical work contains some specific optimizations for crawling news sites. At this
> moment my english writing skills aren't good, but i'll try to represent some interesting aspects of
> my graduate paper somehow=)
>
> On Wed 16 Nov 2011 05:16:05 AM MSK, Lewis John Mcgibbney wrote:
>> ---------- Forwarded message ----------
>> From: Lewis John Mcgibbney<le...@gmail.com>
>> Date: Wed, Nov 16, 2011 at 1:15 AM
>> Subject: Re: Nutch project and my Ph.D. thesis.
>> To: dev@nutch.apache.org
>>
>>
>> Hi Sergey,
>>
>> There was a Professor from somewhere in S America that posted recently
>> rearding some work he did, if you search the archives you may get a taster
>> for work related to Nutch.
>>
>> Also can you provide a link to your work? I would be very intersted in
>> having a look at the areas you have been working on. Also feel free to add
>> your work to the wiki page references for others to see.
>>
>> Thank you.
>>
>> http://wiki.apache.org/nutch/AcademicArticles
>>
>>
>> On Wed, Nov 16, 2011 at 12:39 AM, Sergey A Volkov<sergey.volkov.a@gmail.com
>>> wrote:
>>
>>> Hi!
>>>
>>> I am postgraduate student in Saint Petersburg State University. I was
>>> working with Nutch for about 3 years, have written my graduate work based
>>> on it, and now I don't know what to do in my Ph.D work. (Nobody in my
>>> department (System Programming) deals with web crawling)
>>>
>>> I hope someone knows problems in web crawling, whose solutions can help
>>> Nutch project and me in my future Ph.D. thesis.
>>>
>>> Any ideas?
>>>
>>> Thanks,
>>> Sergey.
>>>
>>
>>
>>
>
>
Re: Fwd: Nutch project and my Ph.D. thesis.
Posted by Sergey A Volkov <se...@gmail.com>.
Thanks!
Unfortunately my work is only in russian - anyway here is the link
https://github.com/volkov/diploma/blob/master/main.pdf?raw=true
Actually it is technical work contains some specific optimizations for
crawling news sites. At this moment my english writing skills aren't
good, but i'll try to represent some interesting aspects of my graduate
paper somehow=)
On Wed 16 Nov 2011 05:16:05 AM MSK, Lewis John Mcgibbney wrote:
> ---------- Forwarded message ----------
> From: Lewis John Mcgibbney<le...@gmail.com>
> Date: Wed, Nov 16, 2011 at 1:15 AM
> Subject: Re: Nutch project and my Ph.D. thesis.
> To: dev@nutch.apache.org
>
>
> Hi Sergey,
>
> There was a Professor from somewhere in S America that posted recently
> rearding some work he did, if you search the archives you may get a taster
> for work related to Nutch.
>
> Also can you provide a link to your work? I would be very intersted in
> having a look at the areas you have been working on. Also feel free to add
> your work to the wiki page references for others to see.
>
> Thank you.
>
> http://wiki.apache.org/nutch/AcademicArticles
>
>
> On Wed, Nov 16, 2011 at 12:39 AM, Sergey A Volkov<sergey.volkov.a@gmail.com
>> wrote:
>
>> Hi!
>>
>> I am postgraduate student in Saint Petersburg State University. I was
>> working with Nutch for about 3 years, have written my graduate work based
>> on it, and now I don't know what to do in my Ph.D work. (Nobody in my
>> department (System Programming) deals with web crawling)
>>
>> I hope someone knows problems in web crawling, whose solutions can help
>> Nutch project and me in my future Ph.D. thesis.
>>
>> Any ideas?
>>
>> Thanks,
>> Sergey.
>>
>
>
>
Fwd: Nutch project and my Ph.D. thesis.
Posted by Lewis John Mcgibbney <le...@gmail.com>.
---------- Forwarded message ----------
From: Lewis John Mcgibbney <le...@gmail.com>
Date: Wed, Nov 16, 2011 at 1:15 AM
Subject: Re: Nutch project and my Ph.D. thesis.
To: dev@nutch.apache.org
Hi Sergey,
There was a Professor from somewhere in S America that posted recently
rearding some work he did, if you search the archives you may get a taster
for work related to Nutch.
Also can you provide a link to your work? I would be very intersted in
having a look at the areas you have been working on. Also feel free to add
your work to the wiki page references for others to see.
Thank you.
http://wiki.apache.org/nutch/AcademicArticles
On Wed, Nov 16, 2011 at 12:39 AM, Sergey A Volkov <sergey.volkov.a@gmail.com
> wrote:
> Hi!
>
> I am postgraduate student in Saint Petersburg State University. I was
> working with Nutch for about 3 years, have written my graduate work based
> on it, and now I don't know what to do in my Ph.D work. (Nobody in my
> department (System Programming) deals with web crawling)
>
> I hope someone knows problems in web crawling, whose solutions can help
> Nutch project and me in my future Ph.D. thesis.
>
> Any ideas?
>
> Thanks,
> Sergey.
>
--
*Lewis*
--
*Lewis*
Re: Nutch project and my Ph.D. thesis.
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Sergey,
There was a Professor from somewhere in S America that posted recently
rearding some work he did, if you search the archives you may get a taster
for work related to Nutch.
Also can you provide a link to your work? I would be very intersted in
having a look at the areas you have been working on. Also feel free to add
your work to the wiki page references for others to see.
Thank you.
http://wiki.apache.org/nutch/AcademicArticles
On Wed, Nov 16, 2011 at 12:39 AM, Sergey A Volkov <sergey.volkov.a@gmail.com
> wrote:
> Hi!
>
> I am postgraduate student in Saint Petersburg State University. I was
> working with Nutch for about 3 years, have written my graduate work based
> on it, and now I don't know what to do in my Ph.D work. (Nobody in my
> department (System Programming) deals with web crawling)
>
> I hope someone knows problems in web crawling, whose solutions can help
> Nutch project and me in my future Ph.D. thesis.
>
> Any ideas?
>
> Thanks,
> Sergey.
>
--
*Lewis*
Re: Nutch project and my Ph.D. thesis.
Posted by Sergey A Volkov <se...@gmail.com>.
On 11/16/2011 04:39 AM, Sergey A Volkov wrote:
> Hi!
>
> I am postgraduate student in Saint Petersburg State University. I was
> working with Nutch for about 3 years, have written my graduate work
> based on it, and now I don't know what to do in my Ph.D work. (Nobody
> in my department (System Programming) deals with web crawling)
>
> I hope someone knows problems in web crawling, whose solutions can
> help Nutch project and me in my future Ph.D. thesis.
>
> Any ideas?
>
> Thanks,
> Sergey.
Sorry!
Wrong list(