You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by John Mendenhall <jo...@surfutopia.net> on 2008/10/21 03:14:50 UTC
nutch parsetext missing for some urls
We are using nutch version nutch-2008-07-22_04-01-29.
We have a crawldb with over 1 million urls.
We have noticed some of the urls in search results
do not have titles. After some research comparing
urls with titles and urls without titles, the urls
without titles have empty parsetext.
Why would some urls have empty parsetext?
Is there some place I can look to determine why
parsetext is missing?
Is the only way to reparse those urls with empty
parsetext to remove the crawl_parse directory for
the corresponding segment and run the nutch parse
command?
Is there something I should do to guarantee all
urls get a parsetext, and hopefully, a title?
Thanks in advance for any assistance or pointers
to other resources or ideas.
JohnM
--
john mendenhall
john@surfutopia.net
surf utopia
internet services
Re: nutch parsetext missing for some urls
Posted by John Mendenhall <jo...@surfutopia.net>.
> Maybe you can reproduce the problem on your environment with URLs publicaly
> available.
>
> What is the mime type for the documents without titles?
Mime type is text/html.
We found the cause of our problem. There was a meta robots
element with content of noindex on the pages without titles.
We have worked through this issue and the titles now exist,
along with the corresponding text.
JohnM
--
john mendenhall
john@surfutopia.net
surf utopia
internet services
Re: nutch parsetext missing for some urls
Posted by Alexander Aristov <al...@gmail.com>.
Maybe you can reproduce the problem on your environment with URLs publicaly
available.
What is the mime type for the documents without titles?
Alexander
2008/10/21 John Mendenhall <jo...@surfutopia.net>
>
> > Can u post some of the urls for which parse text is missing.
>
> I am unable to post the actual urls. This is a private
> project for which exact urls cannot be shared.
>
> JohnM
>
>
>
>
> > On Tue, Oct 21, 2008 at 6:44 AM, John Mendenhall <john@surfutopia.net
> >wrote:
> >
> > > We are using nutch version nutch-2008-07-22_04-01-29.
> > > We have a crawldb with over 1 million urls.
> > >
> > > We have noticed some of the urls in search results
> > > do not have titles. After some research comparing
> > > urls with titles and urls without titles, the urls
> > > without titles have empty parsetext.
> > >
> > > Why would some urls have empty parsetext?
> > > Is there some place I can look to determine why
> > > parsetext is missing?
> > >
> > > Is the only way to reparse those urls with empty
> > > parsetext to remove the crawl_parse directory for
> > > the corresponding segment and run the nutch parse
> > > command?
> > >
> > > Is there something I should do to guarantee all
> > > urls get a parsetext, and hopefully, a title?
> > >
> > > Thanks in advance for any assistance or pointers
> > > to other resources or ideas.
> > >
> > > JohnM
>
> --
> john mendenhall
> john@surfutopia.net
> surf utopia
> internet services
>
--
Best Regards
Alexander Aristov
Re: nutch parsetext missing for some urls
Posted by John Mendenhall <jo...@surfutopia.net>.
> Can u post some of the urls for which parse text is missing.
I am unable to post the actual urls. This is a private
project for which exact urls cannot be shared.
JohnM
> On Tue, Oct 21, 2008 at 6:44 AM, John Mendenhall <jo...@surfutopia.net>wrote:
>
> > We are using nutch version nutch-2008-07-22_04-01-29.
> > We have a crawldb with over 1 million urls.
> >
> > We have noticed some of the urls in search results
> > do not have titles. After some research comparing
> > urls with titles and urls without titles, the urls
> > without titles have empty parsetext.
> >
> > Why would some urls have empty parsetext?
> > Is there some place I can look to determine why
> > parsetext is missing?
> >
> > Is the only way to reparse those urls with empty
> > parsetext to remove the crawl_parse directory for
> > the corresponding segment and run the nutch parse
> > command?
> >
> > Is there something I should do to guarantee all
> > urls get a parsetext, and hopefully, a title?
> >
> > Thanks in advance for any assistance or pointers
> > to other resources or ideas.
> >
> > JohnM
--
john mendenhall
john@surfutopia.net
surf utopia
internet services
Re: nutch parsetext missing for some urls
Posted by Srinivas Gokavarapu <sr...@gmail.com>.
hi,
Can u post some of the urls for which parse text is missing.
On Tue, Oct 21, 2008 at 6:44 AM, John Mendenhall <jo...@surfutopia.net>wrote:
> We are using nutch version nutch-2008-07-22_04-01-29.
> We have a crawldb with over 1 million urls.
>
> We have noticed some of the urls in search results
> do not have titles. After some research comparing
> urls with titles and urls without titles, the urls
> without titles have empty parsetext.
>
> Why would some urls have empty parsetext?
> Is there some place I can look to determine why
> parsetext is missing?
>
> Is the only way to reparse those urls with empty
> parsetext to remove the crawl_parse directory for
> the corresponding segment and run the nutch parse
> command?
>
> Is there something I should do to guarantee all
> urls get a parsetext, and hopefully, a title?
>
> Thanks in advance for any assistance or pointers
> to other resources or ideas.
>
> JohnM
>
> --
> john mendenhall
> john@surfutopia.net
> surf utopia
> internet services
>