You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Sandeep C R <cr...@gmail.com> on 2012/06/11 20:09:39 UTC
Getting seed url
Hello,
I am trying to find a way in which I can get the seed url of current url
being parsed. I have many URL's in seed.txt. I am trying to add additional
metadata for each URL crawled. The metadata depends on the seed URL of the
current URL. This metadata will be later picked by the indexer. I have
written a custom plugin for this purpose. However I am unable to get the
seed url of the current url being parsed.
Ex: This is my seed.txt
http://apache.org
http://amazon.com
http://w3.org
For all URL's crawled for every seed URL, I want to add metadata. The value
of metadata will depend on seed URL. I have a properties file which will
map seed url to metadata value. If seed url is http://apache.org then my
metadata will be something like "project". If it is http://amazon.com then
it will be "estore". I have written a plugin which will add metadata. This
plugin extends HtmlParserFilter. However I am not able find a way to get
the seed url of current url. If http://nutch.apache.org is being parsed
currently, then how do we know the seed url(http:/apache.org) of this url?
Is there any API which I could use in my plugin? Or is there any better way
to achieve this?
Regards,
Sandeep
Re: Getting seed url
Posted by remi tassing <ta...@gmail.com>.
Segments have a field called 'outlinks', could this help?
On Tuesday, June 12, 2012, Sebastian Nagel wrote:
> Hi Sandeep,
>
> tracking the seed(s) for a document could be done by a scoring filter.
> The seed URL must be passed:
> 0 into CrawlDatum's meta by injectedScore()
> (alternatively, use additional fields in the seed file:
> <url> <tab> seed=<url>
> see Injector Javadoc)
> 1 in passScoreBeforeParsing():
> from CrawlDatum to Content
> 2 in passScoreAfterParsing():
> from Content to ParseData
> 3 in distributeScoreToOutlinks():
> from source ParseData to all target/outlink CrawlDatum objects
> 4 in updateDbScore():
> resolve inlinks from multiple seeds
>
> Point 4 shows a little problem: a page may be reachable from multiple
> seeds.
> The web is a graph not a forest of trees each with one seed as root!
>
> Finally: amazon.com is definitely linked from apache.org
> but it is not a "project" site.
> Wouldn't a mapping <domain name> -> <meta data> be more reliable
> (though notoriously incomplete)?
>
> Best,
> Sebastian
>
> On 06/11/2012 08:09 PM, Sandeep C R wrote:
> > Hello,
> >
> > I am trying to find a way in which I can get the seed url of current url
> > being parsed. I have many URL's in seed.txt. I am trying to add
> additional
> > metadata for each URL crawled. The metadata depends on the seed URL of
> the
> > current URL. This metadata will be later picked by the indexer. I have
> > written a custom plugin for this purpose. However I am unable to get the
> > seed url of the current url being parsed.
> >
> > Ex: This is my seed.txt
> >
> > http://apache.org
> > http://amazon.com
> > http://w3.org
> >
> > For all URL's crawled for every seed URL, I want to add metadata. The
> value
> > of metadata will depend on seed URL. I have a properties file which will
> > map seed url to metadata value. If seed url is http://apache.org then my
> > metadata will be something like "project". If it is http://amazon.comthen
> > it will be "estore". I have written a plugin which will add metadata.
> This
> > plugin extends HtmlParserFilter. However I am not able find a way to get
> > the seed url of current url. If http://nutch.apache.org is being parsed
> > currently, then how do we know the seed url(http:/apache.org) of this
> url?
> > Is there any API which I could use in my plugin? Or is there any better
> way
> > to achieve this?
> >
> > Regards,
> > Sandeep
> >
>
>
Re: very long fetch reduce task
Posted by kaveh minooie <ka...@plutoz.com>.
It seems that I've missed a few important details in my original email.
I do apologize for that, but let me clarify a few of them. I see in the
threads and issues that has been created regarding this matter and links
to them has been posted in the previous messages people are usually
talking about encountering errors which is not my case. so here is my
situation. I am running parser inside fetch job (the parsing is enabled
during fetch in the config file) and as markus has pointed out the
parsing seems to be happening during the map tasks not reduce. I do not
experience any time out messages during that very long reduce job. nor
any exceptions or anything else. I don't run out of memory or disk space
either.also that reduce task is not idle either. when i go to the
machine that is running that reduce task and run "top", I can see that
that process is using hundred percent of the cpu (or the core that it is
running on). so in a nut shell, everything seem to be fine except that
it seems to be doing lots and lots of work that at least I can't justify
at this point.
thanks,
On 06/13/2012 10:33 AM, Markus Jelsma wrote:
> In a parsing fetcher iirc outlinks are processed in the mapper (at least when outlinks are followed). If a fetcher's reducer stalls you may run out of memory or disk space.
>
>
> -----Original message-----
>> From:kaveh minooie <ka...@plutoz.com>
>> Sent: Wed 13-Jun-2012 19:28
>> To: user@nutch.apache.org
>> Subject: Re: very long fetch reduce task
>>
>> Thanks for the responses, and yes, in my case, parsing IS enabled and
>> happens during the fetch job.
>>
>> On 06/13/2012 07:43 AM, Ferdy Galema wrote:
>>> You're right. I was already assuming parsing was enabled. If it's not,
>>> normalizing and filtering is most likely the next probable cause why tasks
>>> are stalling.
>>>
>>> On Wed, Jun 13, 2012 at 4:36 PM, Julien Nioche <
>>> lists.digitalpebble@gmail.com> wrote:
>>>
>>>> unless the parsing is activated in the fetch step - this is likely to be a
>>>> different issue e.g. normalization of URL taking forever or something like
>>>> this. Use jstack to see what the problem is
>>>>
>>>> On 13 June 2012 12:36, Ferdy Galema <fe...@kalooga.com> wrote:
>>>>
>>>>> I'd like to add that I've recently opened an issue that describes one of
>>>>> the causes of this problem. Look for the lazy man's profiler trick to see
>>>>> stacktraces of the slow parser task. It will give an indication which
>>>>> parser code is stalling:
>>>>> https://issues.apache.org/jira/browse/NUTCH-1387
>>>>>
>>>>> On Wed, Jun 13, 2012 at 12:40 PM, Lewis John Mcgibbney <
>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>
>>>>>> Hi kaveh,
>>>>>>
>>>>>> We have recently been informed about parsing taking forever and a day
>>>>>> in the reduce phase. This is currently being investigated. FYI the
>>>>>> thread can be found below
>>>>>>
>>>>>> http://www.mail-archive.com/user%40nutch.apache.org/msg06560.html
>>>>>>
>>>>>> I wonder if you have looked into this and if there is a more general
>>>>>> link between such issues?
>>>>>>
>>>>>> Lewis
>>>>>>
>>>>>> On Wed, Jun 13, 2012 at 1:31 AM, kaveh minooie <ka...@plutoz.com>
>>>> wrote:
>>>>>>> Hi everybody
>>>>>>>
>>>>>>> I have an unusual issue. when i run nutch on top off hadoop, after
>>>> the
>>>>>> map
>>>>>>> tasks finish, the reduce task start to finish very fast almost all of
>>>>>> them
>>>>>>> finish in less than 2 hours but there is alway one or two that take a
>>>>> lot
>>>>>>> longer. this is a link to the list of a completed reduce tasks ( that
>>>>> is
>>>>>> all
>>>>>>> of them for that fetch job) and you can see on the list that the last
>>>>> one
>>>>>>> took more than 18 hours to finish and there is another one that took
>>>>> more
>>>>>>> than 6 hours. does any body have any idea why this is happening?
>>>>>>>
>>>>>>> http://plutooz.com/hadoop.html
>>>>>>>
>>>>>>> p.s. this fetch job had about 1.5 million pages in it.
>>>>>>>
>>>>>>> thanks,
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Lewis
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *
>>>> *Open Source Solutions for Text Engineering
>>>>
>>>> http://digitalpebble.blogspot.com/
>>>> http://www.digitalpebble.com
>>>> http://twitter.com/digitalpebble
>>>>
>>>
>>
>> --
>> Kaveh Minooie
>>
>> www.plutoz.com
>>
>>
>>
--
Kaveh Minooie
www.plutoz.com
RE: very long fetch reduce task
Posted by Markus Jelsma <ma...@openindex.io>.
In a parsing fetcher iirc outlinks are processed in the mapper (at least when outlinks are followed). If a fetcher's reducer stalls you may run out of memory or disk space.
-----Original message-----
> From:kaveh minooie <ka...@plutoz.com>
> Sent: Wed 13-Jun-2012 19:28
> To: user@nutch.apache.org
> Subject: Re: very long fetch reduce task
>
> Thanks for the responses, and yes, in my case, parsing IS enabled and
> happens during the fetch job.
>
> On 06/13/2012 07:43 AM, Ferdy Galema wrote:
> > You're right. I was already assuming parsing was enabled. If it's not,
> > normalizing and filtering is most likely the next probable cause why tasks
> > are stalling.
> >
> > On Wed, Jun 13, 2012 at 4:36 PM, Julien Nioche <
> > lists.digitalpebble@gmail.com> wrote:
> >
> >> unless the parsing is activated in the fetch step - this is likely to be a
> >> different issue e.g. normalization of URL taking forever or something like
> >> this. Use jstack to see what the problem is
> >>
> >> On 13 June 2012 12:36, Ferdy Galema <fe...@kalooga.com> wrote:
> >>
> >>> I'd like to add that I've recently opened an issue that describes one of
> >>> the causes of this problem. Look for the lazy man's profiler trick to see
> >>> stacktraces of the slow parser task. It will give an indication which
> >>> parser code is stalling:
> >>> https://issues.apache.org/jira/browse/NUTCH-1387
> >>>
> >>> On Wed, Jun 13, 2012 at 12:40 PM, Lewis John Mcgibbney <
> >>> lewis.mcgibbney@gmail.com> wrote:
> >>>
> >>>> Hi kaveh,
> >>>>
> >>>> We have recently been informed about parsing taking forever and a day
> >>>> in the reduce phase. This is currently being investigated. FYI the
> >>>> thread can be found below
> >>>>
> >>>> http://www.mail-archive.com/user%40nutch.apache.org/msg06560.html
> >>>>
> >>>> I wonder if you have looked into this and if there is a more general
> >>>> link between such issues?
> >>>>
> >>>> Lewis
> >>>>
> >>>> On Wed, Jun 13, 2012 at 1:31 AM, kaveh minooie <ka...@plutoz.com>
> >> wrote:
> >>>>> Hi everybody
> >>>>>
> >>>>> I have an unusual issue. when i run nutch on top off hadoop, after
> >> the
> >>>> map
> >>>>> tasks finish, the reduce task start to finish very fast almost all of
> >>>> them
> >>>>> finish in less than 2 hours but there is alway one or two that take a
> >>> lot
> >>>>> longer. this is a link to the list of a completed reduce tasks ( that
> >>> is
> >>>> all
> >>>>> of them for that fetch job) and you can see on the list that the last
> >>> one
> >>>>> took more than 18 hours to finish and there is another one that took
> >>> more
> >>>>> than 6 hours. does any body have any idea why this is happening?
> >>>>>
> >>>>> http://plutooz.com/hadoop.html
> >>>>>
> >>>>> p.s. this fetch job had about 1.5 million pages in it.
> >>>>>
> >>>>> thanks,
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Lewis
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> *
> >> *Open Source Solutions for Text Engineering
> >>
> >> http://digitalpebble.blogspot.com/
> >> http://www.digitalpebble.com
> >> http://twitter.com/digitalpebble
> >>
> >
>
> --
> Kaveh Minooie
>
> www.plutoz.com
>
>
>
Re: very long fetch reduce task
Posted by kaveh minooie <ka...@plutoz.com>.
Thanks for the responses, and yes, in my case, parsing IS enabled and
happens during the fetch job.
On 06/13/2012 07:43 AM, Ferdy Galema wrote:
> You're right. I was already assuming parsing was enabled. If it's not,
> normalizing and filtering is most likely the next probable cause why tasks
> are stalling.
>
> On Wed, Jun 13, 2012 at 4:36 PM, Julien Nioche <
> lists.digitalpebble@gmail.com> wrote:
>
>> unless the parsing is activated in the fetch step - this is likely to be a
>> different issue e.g. normalization of URL taking forever or something like
>> this. Use jstack to see what the problem is
>>
>> On 13 June 2012 12:36, Ferdy Galema <fe...@kalooga.com> wrote:
>>
>>> I'd like to add that I've recently opened an issue that describes one of
>>> the causes of this problem. Look for the lazy man's profiler trick to see
>>> stacktraces of the slow parser task. It will give an indication which
>>> parser code is stalling:
>>> https://issues.apache.org/jira/browse/NUTCH-1387
>>>
>>> On Wed, Jun 13, 2012 at 12:40 PM, Lewis John Mcgibbney <
>>> lewis.mcgibbney@gmail.com> wrote:
>>>
>>>> Hi kaveh,
>>>>
>>>> We have recently been informed about parsing taking forever and a day
>>>> in the reduce phase. This is currently being investigated. FYI the
>>>> thread can be found below
>>>>
>>>> http://www.mail-archive.com/user%40nutch.apache.org/msg06560.html
>>>>
>>>> I wonder if you have looked into this and if there is a more general
>>>> link between such issues?
>>>>
>>>> Lewis
>>>>
>>>> On Wed, Jun 13, 2012 at 1:31 AM, kaveh minooie <ka...@plutoz.com>
>> wrote:
>>>>> Hi everybody
>>>>>
>>>>> I have an unusual issue. when i run nutch on top off hadoop, after
>> the
>>>> map
>>>>> tasks finish, the reduce task start to finish very fast almost all of
>>>> them
>>>>> finish in less than 2 hours but there is alway one or two that take a
>>> lot
>>>>> longer. this is a link to the list of a completed reduce tasks ( that
>>> is
>>>> all
>>>>> of them for that fetch job) and you can see on the list that the last
>>> one
>>>>> took more than 18 hours to finish and there is another one that took
>>> more
>>>>> than 6 hours. does any body have any idea why this is happening?
>>>>>
>>>>> http://plutooz.com/hadoop.html
>>>>>
>>>>> p.s. this fetch job had about 1.5 million pages in it.
>>>>>
>>>>> thanks,
>>>>
>>>>
>>>>
>>>> --
>>>> Lewis
>>>>
>>>
>>
>>
>>
>> --
>> *
>> *Open Source Solutions for Text Engineering
>>
>> http://digitalpebble.blogspot.com/
>> http://www.digitalpebble.com
>> http://twitter.com/digitalpebble
>>
>
--
Kaveh Minooie
www.plutoz.com
Re: very long fetch reduce task
Posted by Ferdy Galema <fe...@kalooga.com>.
You're right. I was already assuming parsing was enabled. If it's not,
normalizing and filtering is most likely the next probable cause why tasks
are stalling.
On Wed, Jun 13, 2012 at 4:36 PM, Julien Nioche <
lists.digitalpebble@gmail.com> wrote:
> unless the parsing is activated in the fetch step - this is likely to be a
> different issue e.g. normalization of URL taking forever or something like
> this. Use jstack to see what the problem is
>
> On 13 June 2012 12:36, Ferdy Galema <fe...@kalooga.com> wrote:
>
> > I'd like to add that I've recently opened an issue that describes one of
> > the causes of this problem. Look for the lazy man's profiler trick to see
> > stacktraces of the slow parser task. It will give an indication which
> > parser code is stalling:
> > https://issues.apache.org/jira/browse/NUTCH-1387
> >
> > On Wed, Jun 13, 2012 at 12:40 PM, Lewis John Mcgibbney <
> > lewis.mcgibbney@gmail.com> wrote:
> >
> > > Hi kaveh,
> > >
> > > We have recently been informed about parsing taking forever and a day
> > > in the reduce phase. This is currently being investigated. FYI the
> > > thread can be found below
> > >
> > > http://www.mail-archive.com/user%40nutch.apache.org/msg06560.html
> > >
> > > I wonder if you have looked into this and if there is a more general
> > > link between such issues?
> > >
> > > Lewis
> > >
> > > On Wed, Jun 13, 2012 at 1:31 AM, kaveh minooie <ka...@plutoz.com>
> wrote:
> > > > Hi everybody
> > > >
> > > > I have an unusual issue. when i run nutch on top off hadoop, after
> the
> > > map
> > > > tasks finish, the reduce task start to finish very fast almost all of
> > > them
> > > > finish in less than 2 hours but there is alway one or two that take a
> > lot
> > > > longer. this is a link to the list of a completed reduce tasks ( that
> > is
> > > all
> > > > of them for that fetch job) and you can see on the list that the last
> > one
> > > > took more than 18 hours to finish and there is another one that took
> > more
> > > > than 6 hours. does any body have any idea why this is happening?
> > > >
> > > > http://plutooz.com/hadoop.html
> > > >
> > > > p.s. this fetch job had about 1.5 million pages in it.
> > > >
> > > > thanks,
> > >
> > >
> > >
> > > --
> > > Lewis
> > >
> >
>
>
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
Re: very long fetch reduce task
Posted by Julien Nioche <li...@gmail.com>.
unless the parsing is activated in the fetch step - this is likely to be a
different issue e.g. normalization of URL taking forever or something like
this. Use jstack to see what the problem is
On 13 June 2012 12:36, Ferdy Galema <fe...@kalooga.com> wrote:
> I'd like to add that I've recently opened an issue that describes one of
> the causes of this problem. Look for the lazy man's profiler trick to see
> stacktraces of the slow parser task. It will give an indication which
> parser code is stalling:
> https://issues.apache.org/jira/browse/NUTCH-1387
>
> On Wed, Jun 13, 2012 at 12:40 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
> > Hi kaveh,
> >
> > We have recently been informed about parsing taking forever and a day
> > in the reduce phase. This is currently being investigated. FYI the
> > thread can be found below
> >
> > http://www.mail-archive.com/user%40nutch.apache.org/msg06560.html
> >
> > I wonder if you have looked into this and if there is a more general
> > link between such issues?
> >
> > Lewis
> >
> > On Wed, Jun 13, 2012 at 1:31 AM, kaveh minooie <ka...@plutoz.com> wrote:
> > > Hi everybody
> > >
> > > I have an unusual issue. when i run nutch on top off hadoop, after the
> > map
> > > tasks finish, the reduce task start to finish very fast almost all of
> > them
> > > finish in less than 2 hours but there is alway one or two that take a
> lot
> > > longer. this is a link to the list of a completed reduce tasks ( that
> is
> > all
> > > of them for that fetch job) and you can see on the list that the last
> one
> > > took more than 18 hours to finish and there is another one that took
> more
> > > than 6 hours. does any body have any idea why this is happening?
> > >
> > > http://plutooz.com/hadoop.html
> > >
> > > p.s. this fetch job had about 1.5 million pages in it.
> > >
> > > thanks,
> >
> >
> >
> > --
> > Lewis
> >
>
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
Re: very long fetch reduce task
Posted by Ferdy Galema <fe...@kalooga.com>.
I'd like to add that I've recently opened an issue that describes one of
the causes of this problem. Look for the lazy man's profiler trick to see
stacktraces of the slow parser task. It will give an indication which
parser code is stalling:
https://issues.apache.org/jira/browse/NUTCH-1387
On Wed, Jun 13, 2012 at 12:40 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Hi kaveh,
>
> We have recently been informed about parsing taking forever and a day
> in the reduce phase. This is currently being investigated. FYI the
> thread can be found below
>
> http://www.mail-archive.com/user%40nutch.apache.org/msg06560.html
>
> I wonder if you have looked into this and if there is a more general
> link between such issues?
>
> Lewis
>
> On Wed, Jun 13, 2012 at 1:31 AM, kaveh minooie <ka...@plutoz.com> wrote:
> > Hi everybody
> >
> > I have an unusual issue. when i run nutch on top off hadoop, after the
> map
> > tasks finish, the reduce task start to finish very fast almost all of
> them
> > finish in less than 2 hours but there is alway one or two that take a lot
> > longer. this is a link to the list of a completed reduce tasks ( that is
> all
> > of them for that fetch job) and you can see on the list that the last one
> > took more than 18 hours to finish and there is another one that took more
> > than 6 hours. does any body have any idea why this is happening?
> >
> > http://plutooz.com/hadoop.html
> >
> > p.s. this fetch job had about 1.5 million pages in it.
> >
> > thanks,
>
>
>
> --
> Lewis
>
Re: very long fetch reduce task
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi kaveh,
We have recently been informed about parsing taking forever and a day
in the reduce phase. This is currently being investigated. FYI the
thread can be found below
http://www.mail-archive.com/user%40nutch.apache.org/msg06560.html
I wonder if you have looked into this and if there is a more general
link between such issues?
Lewis
On Wed, Jun 13, 2012 at 1:31 AM, kaveh minooie <ka...@plutoz.com> wrote:
> Hi everybody
>
> I have an unusual issue. when i run nutch on top off hadoop, after the map
> tasks finish, the reduce task start to finish very fast almost all of them
> finish in less than 2 hours but there is alway one or two that take a lot
> longer. this is a link to the list of a completed reduce tasks ( that is all
> of them for that fetch job) and you can see on the list that the last one
> took more than 18 hours to finish and there is another one that took more
> than 6 hours. does any body have any idea why this is happening?
>
> http://plutooz.com/hadoop.html
>
> p.s. this fetch job had about 1.5 million pages in it.
>
> thanks,
--
Lewis
very long fetch reduce task
Posted by kaveh minooie <ka...@plutoz.com>.
Hi everybody
I have an unusual issue. when i run nutch on top off hadoop, after the
map tasks finish, the reduce task start to finish very fast almost all
of them finish in less than 2 hours but there is alway one or two that
take a lot longer. this is a link to the list of a completed reduce
tasks ( that is all of them for that fetch job) and you can see on the
list that the last one took more than 18 hours to finish and there is
another one that took more than 6 hours. does any body have any idea why
this is happening?
http://plutooz.com/hadoop.html
p.s. this fetch job had about 1.5 million pages in it.
thanks,
Re: Getting seed url
Posted by Sebastian Nagel <wa...@googlemail.com>.
Thanks Julien,
I've missed that urlmeta passes the tags to the outlinks.
Sebastian
On 06/12/2012 03:42 PM, Julien Nioche wrote:
> forgot to say : this would work by adding a seed metadata to the urls in
> the seed list, the value of which is then propagated by the scoring filter
> in urlmeta
>
> On 12 June 2012 14:41, Julien Nioche <li...@gmail.com> wrote:
>
>> That's the idea indeed. The urlmeta plugin allows to do that simply by
>> setting urlmeta.tags in nutch-site.xml (see nutch-default.xml for
>> description etc...)
>>
>>
>>
>> On 11 June 2012 22:45, Sebastian Nagel <wa...@googlemail.com> wrote:
>>
>>> Hi Sandeep,
>>>
>>> tracking the seed(s) for a document could be done by a scoring filter.
>>> The seed URL must be passed:
>>> 0 into CrawlDatum's meta by injectedScore()
>>> (alternatively, use additional fields in the seed file:
>>> <url> <tab> seed=<url>
>>> see Injector Javadoc)
>>> 1 in passScoreBeforeParsing():
>>> from CrawlDatum to Content
>>> 2 in passScoreAfterParsing():
>>> from Content to ParseData
>>> 3 in distributeScoreToOutlinks():
>>> from source ParseData to all target/outlink CrawlDatum objects
>>> 4 in updateDbScore():
>>> resolve inlinks from multiple seeds
>>>
>>> Point 4 shows a little problem: a page may be reachable from multiple
>>> seeds.
>>> The web is a graph not a forest of trees each with one seed as root!
>>>
>>> Finally: amazon.com is definitely linked from apache.org
>>> but it is not a "project" site.
>>> Wouldn't a mapping <domain name> -> <meta data> be more reliable
>>> (though notoriously incomplete)?
>>>
>>> Best,
>>> Sebastian
>>>
>>> On 06/11/2012 08:09 PM, Sandeep C R wrote:
>>>> Hello,
>>>>
>>>> I am trying to find a way in which I can get the seed url of current url
>>>> being parsed. I have many URL's in seed.txt. I am trying to add
>>> additional
>>>> metadata for each URL crawled. The metadata depends on the seed URL of
>>> the
>>>> current URL. This metadata will be later picked by the indexer. I have
>>>> written a custom plugin for this purpose. However I am unable to get the
>>>> seed url of the current url being parsed.
>>>>
>>>> Ex: This is my seed.txt
>>>>
>>>> http://apache.org
>>>> http://amazon.com
>>>> http://w3.org
>>>>
>>>> For all URL's crawled for every seed URL, I want to add metadata. The
>>> value
>>>> of metadata will depend on seed URL. I have a properties file which will
>>>> map seed url to metadata value. If seed url is http://apache.org then
>>> my
>>>> metadata will be something like "project". If it is http://amazon.comthen
>>>> it will be "estore". I have written a plugin which will add metadata.
>>> This
>>>> plugin extends HtmlParserFilter. However I am not able find a way to get
>>>> the seed url of current url. If http://nutch.apache.org is being parsed
>>>> currently, then how do we know the seed url(http:/apache.org) of this
>>> url?
>>>> Is there any API which I could use in my plugin? Or is there any better
>>> way
>>>> to achieve this?
>>>>
>>>> Regards,
>>>> Sandeep
>>>>
>>>
>>>
>>
>>
>> --
>> *
>> *Open Source Solutions for Text Engineering
>>
>> http://digitalpebble.blogspot.com/
>> http://www.digitalpebble.com
>> http://twitter.com/digitalpebble
>>
>>
>
>
Re: Getting seed url
Posted by Julien Nioche <li...@gmail.com>.
forgot to say : this would work by adding a seed metadata to the urls in
the seed list, the value of which is then propagated by the scoring filter
in urlmeta
On 12 June 2012 14:41, Julien Nioche <li...@gmail.com> wrote:
> That's the idea indeed. The urlmeta plugin allows to do that simply by
> setting urlmeta.tags in nutch-site.xml (see nutch-default.xml for
> description etc...)
>
>
>
> On 11 June 2012 22:45, Sebastian Nagel <wa...@googlemail.com> wrote:
>
>> Hi Sandeep,
>>
>> tracking the seed(s) for a document could be done by a scoring filter.
>> The seed URL must be passed:
>> 0 into CrawlDatum's meta by injectedScore()
>> (alternatively, use additional fields in the seed file:
>> <url> <tab> seed=<url>
>> see Injector Javadoc)
>> 1 in passScoreBeforeParsing():
>> from CrawlDatum to Content
>> 2 in passScoreAfterParsing():
>> from Content to ParseData
>> 3 in distributeScoreToOutlinks():
>> from source ParseData to all target/outlink CrawlDatum objects
>> 4 in updateDbScore():
>> resolve inlinks from multiple seeds
>>
>> Point 4 shows a little problem: a page may be reachable from multiple
>> seeds.
>> The web is a graph not a forest of trees each with one seed as root!
>>
>> Finally: amazon.com is definitely linked from apache.org
>> but it is not a "project" site.
>> Wouldn't a mapping <domain name> -> <meta data> be more reliable
>> (though notoriously incomplete)?
>>
>> Best,
>> Sebastian
>>
>> On 06/11/2012 08:09 PM, Sandeep C R wrote:
>> > Hello,
>> >
>> > I am trying to find a way in which I can get the seed url of current url
>> > being parsed. I have many URL's in seed.txt. I am trying to add
>> additional
>> > metadata for each URL crawled. The metadata depends on the seed URL of
>> the
>> > current URL. This metadata will be later picked by the indexer. I have
>> > written a custom plugin for this purpose. However I am unable to get the
>> > seed url of the current url being parsed.
>> >
>> > Ex: This is my seed.txt
>> >
>> > http://apache.org
>> > http://amazon.com
>> > http://w3.org
>> >
>> > For all URL's crawled for every seed URL, I want to add metadata. The
>> value
>> > of metadata will depend on seed URL. I have a properties file which will
>> > map seed url to metadata value. If seed url is http://apache.org then
>> my
>> > metadata will be something like "project". If it is http://amazon.comthen
>> > it will be "estore". I have written a plugin which will add metadata.
>> This
>> > plugin extends HtmlParserFilter. However I am not able find a way to get
>> > the seed url of current url. If http://nutch.apache.org is being parsed
>> > currently, then how do we know the seed url(http:/apache.org) of this
>> url?
>> > Is there any API which I could use in my plugin? Or is there any better
>> way
>> > to achieve this?
>> >
>> > Regards,
>> > Sandeep
>> >
>>
>>
>
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
>
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
Re: Getting seed url
Posted by Julien Nioche <li...@gmail.com>.
That's the idea indeed. The urlmeta plugin allows to do that simply by
setting urlmeta.tags in nutch-site.xml (see nutch-default.xml for
description etc...)
On 11 June 2012 22:45, Sebastian Nagel <wa...@googlemail.com> wrote:
> Hi Sandeep,
>
> tracking the seed(s) for a document could be done by a scoring filter.
> The seed URL must be passed:
> 0 into CrawlDatum's meta by injectedScore()
> (alternatively, use additional fields in the seed file:
> <url> <tab> seed=<url>
> see Injector Javadoc)
> 1 in passScoreBeforeParsing():
> from CrawlDatum to Content
> 2 in passScoreAfterParsing():
> from Content to ParseData
> 3 in distributeScoreToOutlinks():
> from source ParseData to all target/outlink CrawlDatum objects
> 4 in updateDbScore():
> resolve inlinks from multiple seeds
>
> Point 4 shows a little problem: a page may be reachable from multiple
> seeds.
> The web is a graph not a forest of trees each with one seed as root!
>
> Finally: amazon.com is definitely linked from apache.org
> but it is not a "project" site.
> Wouldn't a mapping <domain name> -> <meta data> be more reliable
> (though notoriously incomplete)?
>
> Best,
> Sebastian
>
> On 06/11/2012 08:09 PM, Sandeep C R wrote:
> > Hello,
> >
> > I am trying to find a way in which I can get the seed url of current url
> > being parsed. I have many URL's in seed.txt. I am trying to add
> additional
> > metadata for each URL crawled. The metadata depends on the seed URL of
> the
> > current URL. This metadata will be later picked by the indexer. I have
> > written a custom plugin for this purpose. However I am unable to get the
> > seed url of the current url being parsed.
> >
> > Ex: This is my seed.txt
> >
> > http://apache.org
> > http://amazon.com
> > http://w3.org
> >
> > For all URL's crawled for every seed URL, I want to add metadata. The
> value
> > of metadata will depend on seed URL. I have a properties file which will
> > map seed url to metadata value. If seed url is http://apache.org then my
> > metadata will be something like "project". If it is http://amazon.comthen
> > it will be "estore". I have written a plugin which will add metadata.
> This
> > plugin extends HtmlParserFilter. However I am not able find a way to get
> > the seed url of current url. If http://nutch.apache.org is being parsed
> > currently, then how do we know the seed url(http:/apache.org) of this
> url?
> > Is there any API which I could use in my plugin? Or is there any better
> way
> > to achieve this?
> >
> > Regards,
> > Sandeep
> >
>
>
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
Re: Getting seed url
Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi Sandeep,
tracking the seed(s) for a document could be done by a scoring filter.
The seed URL must be passed:
0 into CrawlDatum's meta by injectedScore()
(alternatively, use additional fields in the seed file:
<url> <tab> seed=<url>
see Injector Javadoc)
1 in passScoreBeforeParsing():
from CrawlDatum to Content
2 in passScoreAfterParsing():
from Content to ParseData
3 in distributeScoreToOutlinks():
from source ParseData to all target/outlink CrawlDatum objects
4 in updateDbScore():
resolve inlinks from multiple seeds
Point 4 shows a little problem: a page may be reachable from multiple seeds.
The web is a graph not a forest of trees each with one seed as root!
Finally: amazon.com is definitely linked from apache.org
but it is not a "project" site.
Wouldn't a mapping <domain name> -> <meta data> be more reliable
(though notoriously incomplete)?
Best,
Sebastian
On 06/11/2012 08:09 PM, Sandeep C R wrote:
> Hello,
>
> I am trying to find a way in which I can get the seed url of current url
> being parsed. I have many URL's in seed.txt. I am trying to add additional
> metadata for each URL crawled. The metadata depends on the seed URL of the
> current URL. This metadata will be later picked by the indexer. I have
> written a custom plugin for this purpose. However I am unable to get the
> seed url of the current url being parsed.
>
> Ex: This is my seed.txt
>
> http://apache.org
> http://amazon.com
> http://w3.org
>
> For all URL's crawled for every seed URL, I want to add metadata. The value
> of metadata will depend on seed URL. I have a properties file which will
> map seed url to metadata value. If seed url is http://apache.org then my
> metadata will be something like "project". If it is http://amazon.com then
> it will be "estore". I have written a plugin which will add metadata. This
> plugin extends HtmlParserFilter. However I am not able find a way to get
> the seed url of current url. If http://nutch.apache.org is being parsed
> currently, then how do we know the seed url(http:/apache.org) of this url?
> Is there any API which I could use in my plugin? Or is there any better way
> to achieve this?
>
> Regards,
> Sandeep
>