You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Pravin Agrawal <Pr...@persistent.co.in> on 2012/04/06 10:50:05 UTC
Question related to NUCTH 1044 redirected URLS and invalid scores
Hi All,
While working with latest release nutch 1.4, I encountered a URL where the homepage itself was redirected to some other URL and each outlinks from that pages were given a score of 0.
Found the related issue in jira (NUCTH-1044) which says it is fixed in nutch 1.4.
But While I was going through the patch attached, I observed that there are four files that were to be changed, but the changes committed in revision 1156342 shows only two files checked in.
Latest Source code for 1.4 is also missing on the changes for OPICScoringFIlter and LinkAnalysisScoringFilter given in the patch.
can anyone please clarify on this?
Is the attached patch in NUTCH 104 jira updated?
Can I apply the patch without worrying about the side effects?
Thanks,
Pravin
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Question related to NUCTH 1044 redirected URLS and invalid scores
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Pravin,
I won't have time until the weekend to get around to this.
I'll try my best though when the time comes around.
On Tue, Apr 24, 2012 at 4:19 PM, Pravin Agrawal
<Pr...@persistent.co.in> wrote:
> Hi Lewis, thanks for the reply. Sorry I couldn't get back to you soon as I was on vacation.
>
>
>
> I tried out the NUTCH 1044 patch on nutch 1.4 with a test website where a jsp page sends a 302 redirect request to another jsp page. But here I observed that the score of redirected URL is still set to 0 in crawl datum, whereas according to the patch it should be 1. Here the crawl dump
>
>
>
> http://lcoalhost/Test/302_Redirect.jsp Version: 7
>
> Status: 4 (db_redir_temp)
>
> Fetch time: Thu May 24 19:45:47 IST 2012
>
> Modified time: Tue Apr 24 19:45:47 IST 2012
>
> Retries since fetch: 0
>
> Retry interval: 2592000 seconds (30 days)
>
> Score: 1.0
>
> Signature: null
>
> Metadata: _pst_: temp_moved(13), lastModified=0: http://localhost/Test/302_Redirect1.jsp
>
>
>
> http://localhost/Test/302_Redirect1.jsp Version: 7
>
> Status: 2 (db_fetched)
>
> Fetch time: Thu May 24 19:45:48 IST 2012
>
> Modified time: Tue Apr 24 19:45:48 IST 2012
>
> Retries since fetch: 0
>
> Retry interval: 2592000 seconds (30 days)
>
> Score: 0.0
>
> Signature: 8fd921e73ae1bdd60fc509fba24548a5
>
> Metadata: _pst_: success(1), lastModified=0_repr_: http://localhost/Test/302_Redirect.jsp
>
>
>
> I also carried out the above test with 301 redirected urls and soft redirected url, but the result are same as the one without patch.
>
>
>
> Can you please verify this at your end and let me know.
>
>
>
> Thanks,
>
> Pravin
>
> ________________________________________
>
> From: Lewis John Mcgibbney [lewis.mcgibbney@gmail.com]
>
> Sent: Friday, April 06, 2012 8:12 PM
>
> To: user@nutch.apache.org
>
> Subject: Re: Question related to NUCTH 1044 redirected URLS and invalid scores
>
>
>
> Hi Pravin,
>
>
>
> By the looks of it, the commit should have included changes to 4 files as
>
> you state and as the patch shows.
>
>
>
> Maybe we need to revisit this one and make a commit to trunk?
>
>
>
> I would apply the patch then try to reproduce the incorrect results and see
>
> where you get.
>
>
>
> Please let us know.
>
>
>
> lewis
>
>
>
> On Fri, Apr 6, 2012 at 9:50 AM, Pravin Agrawal <
>
> Pravin_Agrawal@persistent.co.in> wrote:
>
>
>
>> Hi All,
>
>>
>
>> While working with latest release nutch 1.4, I encountered a URL where the
>
>> homepage itself was redirected to some other URL and each outlinks from
>
>> that pages were given a score of 0.
>
>> Found the related issue in jira (NUCTH-1044) which says it is fixed in
>
>> nutch 1.4.
>
>> But While I was going through the patch attached, I observed that there
>
>> are four files that were to be changed, but the changes committed in
>
>> revision 1156342 shows only two files checked in.
>
>> Latest Source code for 1.4 is also missing on the changes for
>
>> OPICScoringFIlter and LinkAnalysisScoringFilter given in the patch.
>
>> can anyone please clarify on this?
>
>> Is the attached patch in NUTCH 104 jira updated?
>
>> Can I apply the patch without worrying about the side effects?
>
>>
>
>> Thanks,
>
>> Pravin
>
>>
>
>> DISCLAIMER
>
>> ==========
>
>> This e-mail may contain privileged and confidential information which is
>
>> the property of Persistent Systems Ltd. It is intended only for the use of
>
>> the individual or entity to which it is addressed. If you are not the
>
>> intended recipient, you are not authorized to read, retain, copy, print,
>
>> distribute or use this message. If you have received this communication in
>
>> error, please notify the sender and delete all copies of this message.
>
>> Persistent Systems Ltd. does not accept any liability for virus infected
>
>> mails.
>
>>
>
>
>
>
>
>
>
> --
>
> *Lewis*
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
--
Lewis
RE: Question related to NUCTH 1044 redirected URLS and invalid
scores
Posted by Pravin Agrawal <Pr...@persistent.co.in>.
Hi Lewis, thanks for the reply. Sorry I couldn't get back to you soon as I was on vacation.
I tried out the NUTCH 1044 patch on nutch 1.4 with a test website where a jsp page sends a 302 redirect request to another jsp page. But here I observed that the score of redirected URL is still set to 0 in crawl datum, whereas according to the patch it should be 1. Here the crawl dump
http://lcoalhost/Test/302_Redirect.jsp Version: 7
Status: 4 (db_redir_temp)
Fetch time: Thu May 24 19:45:47 IST 2012
Modified time: Tue Apr 24 19:45:47 IST 2012
Retries since fetch: 0
Retry interval: 2592000 seconds (30 days)
Score: 1.0
Signature: null
Metadata: _pst_: temp_moved(13), lastModified=0: http://localhost/Test/302_Redirect1.jsp
http://localhost/Test/302_Redirect1.jsp Version: 7
Status: 2 (db_fetched)
Fetch time: Thu May 24 19:45:48 IST 2012
Modified time: Tue Apr 24 19:45:48 IST 2012
Retries since fetch: 0
Retry interval: 2592000 seconds (30 days)
Score: 0.0
Signature: 8fd921e73ae1bdd60fc509fba24548a5
Metadata: _pst_: success(1), lastModified=0_repr_: http://localhost/Test/302_Redirect.jsp
I also carried out the above test with 301 redirected urls and soft redirected url, but the result are same as the one without patch.
Can you please verify this at your end and let me know.
Thanks,
Pravin
________________________________________
From: Lewis John Mcgibbney [lewis.mcgibbney@gmail.com]
Sent: Friday, April 06, 2012 8:12 PM
To: user@nutch.apache.org
Subject: Re: Question related to NUCTH 1044 redirected URLS and invalid scores
Hi Pravin,
By the looks of it, the commit should have included changes to 4 files as
you state and as the patch shows.
Maybe we need to revisit this one and make a commit to trunk?
I would apply the patch then try to reproduce the incorrect results and see
where you get.
Please let us know.
lewis
On Fri, Apr 6, 2012 at 9:50 AM, Pravin Agrawal <
Pravin_Agrawal@persistent.co.in> wrote:
> Hi All,
>
> While working with latest release nutch 1.4, I encountered a URL where the
> homepage itself was redirected to some other URL and each outlinks from
> that pages were given a score of 0.
> Found the related issue in jira (NUCTH-1044) which says it is fixed in
> nutch 1.4.
> But While I was going through the patch attached, I observed that there
> are four files that were to be changed, but the changes committed in
> revision 1156342 shows only two files checked in.
> Latest Source code for 1.4 is also missing on the changes for
> OPICScoringFIlter and LinkAnalysisScoringFilter given in the patch.
> can anyone please clarify on this?
> Is the attached patch in NUTCH 104 jira updated?
> Can I apply the patch without worrying about the side effects?
>
> Thanks,
> Pravin
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
--
*Lewis*
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Question related to NUCTH 1044 redirected URLS and invalid scores
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Pravin,
By the looks of it, the commit should have included changes to 4 files as
you state and as the patch shows.
Maybe we need to revisit this one and make a commit to trunk?
I would apply the patch then try to reproduce the incorrect results and see
where you get.
Please let us know.
lewis
On Fri, Apr 6, 2012 at 9:50 AM, Pravin Agrawal <
Pravin_Agrawal@persistent.co.in> wrote:
> Hi All,
>
> While working with latest release nutch 1.4, I encountered a URL where the
> homepage itself was redirected to some other URL and each outlinks from
> that pages were given a score of 0.
> Found the related issue in jira (NUCTH-1044) which says it is fixed in
> nutch 1.4.
> But While I was going through the patch attached, I observed that there
> are four files that were to be changed, but the changes committed in
> revision 1156342 shows only two files checked in.
> Latest Source code for 1.4 is also missing on the changes for
> OPICScoringFIlter and LinkAnalysisScoringFilter given in the patch.
> can anyone please clarify on this?
> Is the attached patch in NUTCH 104 jira updated?
> Can I apply the patch without worrying about the side effects?
>
> Thanks,
> Pravin
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
--
*Lewis*