You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Andrzej Bialecki <ab...@getopt.org> on 2007/03/03 01:05:33 UTC
Issues pending before 0.9 release
Hi all,
The following issues need to be discussed and appropriate action taken
before the 0.9 release:
Blocker
========
* NUTCH-400 (Update & add missing license headers) - I believe this is
fixed and should be closed
* NUTCH-353 (pages that serverside forwards will be refetched every
time) - this was partially fixed in NUTCH-273, but a more complete
solution would require significant changes to LinkDb. As there are no
patches implementing this, I left it open, but it's no longer as
critical as it was before. I propose to move it to "Major" and address
it in the next release.
* NUTCH-233 (wrong regular expression hang reduce process for ever) - I
propose to apply the fix provided by Sean Dean and close this issue for now.
Critical
========
* NUTCH-436 (Incorrect handling of relative paths when the embedded URL
path is empty). There is no patch available yet. If someone could
contribute a patch I'd like to see this fixed before the release.
* NUTCH-427 (protocol-smb). This relies on a LGPL library, and it's
certainly not critical (as this is an optional new feature). I propose
to change it to Major, and make a decision - do we want another plugin
like parse-mp3 or parse-rtf, or not.
* NUTCH-381 (Ignore external link not work as expected) - I'll try to
reproduce it, and if I find an easy fix I'd like to apply it before the
release.
* NUTCH-277 (Fetcher dies because of "max. redirects") - I wasn't able
to reproduce it. If there is no updated information on this I propose to
close it with "Can't reproduce".
* NUTCH-167 (Observation of <META NAME="ROBOTS" CONTENT="NOARCHIVE">) -
there's a patch which I tested in a limited production env. If there are
no objections I'd like to apply it before the release.
Major
=====
There are 84 major issues, but some of them are either invalid, or
should be "minor", or no longer apply and should be closed. Please
review them if you can and provide some comments or recommendations if
you think you have some new information.
One decision also that we need to make is which version of Hadoop should
be included in the release. Current trunk uses 0.10.1, I have a set of
production-tested patches that use 0.11.2, and today the Hadoop team
released 0.12.0 (to be followed shortly by a 0.12.1, most likely in time
before our release). The most conservative option is to stay with
0.10.1, but by the time people start using Nutch this will be a fairly
old version already. I propose to upgrade to 0.11.2. We could use 0.12.1
- but in this case with the expectation that we release less than stable
version of Nutch to be soon followed by a minor stable release ...
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Issues pending before 0.9 release
Posted by Andrzej Bialecki <ab...@getopt.org>.
rubdabadub wrote:
> On 3/22/07, Andrzej Bialecki <ab...@getopt.org> wrote:
>> rubdabadub wrote:
>> > Hi:
>> >
>> > Just wondering about NUTCH-61
>> >
>> > http://issues.apache.org/jira/browse/Nutch-61
>> >
>> > Will it make the 0.9 cut?
>> >
>> > It would be nice if it did. Its probably too late.
>>
>> This was discussed before - it will be applied right after the release.
>
> Hello Andrzej:
>
> Please provide some kind love to Nutch-61 :-)
:) Yes, that's the next thing I'm going to do whenever I get some free time.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Issues pending before 0.9 release
Posted by rubdabadub <ru...@gmail.com>.
On 3/22/07, Andrzej Bialecki <ab...@getopt.org> wrote:
> rubdabadub wrote:
> > Hi:
> >
> > Just wondering about NUTCH-61
> >
> > http://issues.apache.org/jira/browse/Nutch-61
> >
> > Will it make the 0.9 cut?
> >
> > It would be nice if it did. Its probably too late.
>
> This was discussed before - it will be applied right after the release.
Hello Andrzej:
Please provide some kind love to Nutch-61 :-)
I would be very usefull. Thank you for your kind attention.
Regards
Rajesh.
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
Re: Issues pending before 0.9 release
Posted by Andrzej Bialecki <ab...@getopt.org>.
rubdabadub wrote:
> Hi:
>
> Just wondering about NUTCH-61
>
> http://issues.apache.org/jira/browse/Nutch-61
>
> Will it make the 0.9 cut?
>
> It would be nice if it did. Its probably too late.
This was discussed before - it will be applied right after the release.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Issues pending before 0.9 release
Posted by rubdabadub <ru...@gmail.com>.
Hi:
Just wondering about NUTCH-61
http://issues.apache.org/jira/browse/Nutch-61
Will it make the 0.9 cut?
It would be nice if it did. Its probably too late.
Regards
On 3/21/07, Dennis Kubes <nu...@dragonflymc.com> wrote:
> I am good to go as well.
>
> Dennis Kubes
>
> Andrzej Bialecki wrote:
> > Sami Siren wrote:
> >> Andrzej Bialecki wrote:
> >>> Hi all,
> >>>
> >>> I just committed Hadoop 0.12.1. Let's double-check that it works ok.
> >>> Here's the list of Critical/Blocker issues I mentioned before, and their
> >>> current status:
> >>>
> >>> Any other stuff we need to fix before the release?
> >>
> >> I am satisfied except the broken bin/nutch.
> >
> > Fixed now - tested both under Cygwin and Fedora.
> >
>
Re: Issues pending before 0.9 release
Posted by Dennis Kubes <nu...@dragonflymc.com>.
I worked through this swf issue a little more and it seems that java 6
parses out the content differently than java 5. My guess is that it is
some type of collection change from 5 to 6 because it looks like only
the ordering of the elements is different.
Dennis Kubes
Sample
-------- Help javascript:openCrosslinkWindow('/go/adobeacquisition')
Macromedia Home /go/gnav_search?loc=en_us MovieClip solutions
/go/gnav_showcase _sans rollOut To ensure the best possible Internet
Experience, please download the latest version of the free
/go/gnav_store International Products devnet en_us /go/gnav_products
AppleGothic Macromedia Flash Player active products String Store
downloads rollOver Adobe Home /go/gnav_your_account /go/gnav_downloads
Showcase bluePill /go/gnav_company /go/gnav_support /go/gnav_help
javascript:openCrosslinkWindow('/go/gnav_adobe_home') home Home Array
/go/gnav_fl_minmessage textColor Developers Support color support
showcase button /go/gnav_mm_home tabHolder selected Solutions
LocaleManager Verdana /go/gnav_devnet Acquisition Info /go/gnav_cart
Company /go/gnav_solutions company Downloads TextFormat
Java 6
-------- tabHolder LocaleManager Downloads /go/gnav_mm_home AppleGothic
downloads MovieClip Acquisition Info rollOut _sans Home String active
Macromedia Home Store /go/gnav_company /go/gnav_products color
javascript:openCrosslinkWindow('/go/gnav_adobe_home') Adobe Home button
support home javascript:openCrosslinkWindow('/go/adobeacquisition')
products /go/gnav_store /go/gnav_your_account Help selected
/go/gnav_help bluePill Macromedia Flash Player Array en_us Solutions
International /go/gnav_solutions TextFormat /go/gnav_search?loc=en_us
Company /go/gnav_showcase To ensure the best possible Internet
Experience, please download the latest version of the free /go/gnav_cart
/go/gnav_devnet rollOver textColor devnet /go/gnav_support Products
solutions Developers Verdana Showcase /go/gnav_fl_minmessage company
/go/gnav_downloads Support showcase
Andrzej Bialecki wrote:
> Dennis Kubes wrote:
>> I did an update, clean, and test and go no errors.
>>
>> BUILD SUCCESSFUL
>> Total time: 6 minutes
>
> It seems this is related to JDK 1.6 - when I switched back to 1.5 all
> tests passed successfully, switching again to 1.6 causes the parse-swf
> test to fail. I'm not sure what is the reason - it seems that the
> results of text extraction are completely different under 1.6 ...
>
Re: Issues pending before 0.9 release
Posted by Andrzej Bialecki <ab...@getopt.org>.
Dennis Kubes wrote:
> I did an update, clean, and test and go no errors.
>
> BUILD SUCCESSFUL
> Total time: 6 minutes
It seems this is related to JDK 1.6 - when I switched back to 1.5 all
tests passed successfully, switching again to 1.6 causes the parse-swf
test to fail. I'm not sure what is the reason - it seems that the
results of text extraction are completely different under 1.6 ...
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Issues pending before 0.9 release
Posted by Andrzej Bialecki <ab...@getopt.org>.
Sami Siren wrote:
>
>> Let's make it the best release ever! :)
>
> I have a good feeling about this one. There's some nice marketing
> material about crawling efficiency [1]. I should probably extend
> benching to indexing and searching too.
>
> [1] http://blog.foofactory.fi/2007/03/twice-speed-half-size.html
Yes, I saw this - great stuff :)
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Issues pending before 0.9 release
Posted by Sami Siren <ss...@gmail.com>.
Andrzej Bialecki wrote:
> I can't figure out what's wrong with the SWF parser when used with JDK
> 1.6, it works just fine with 1.5 .. However, I propose to add a release
> note somewhere that warns about this, and move on with the release anyway.
+1
A jira issue is probably enough.
> If there are no further issues (anyone?), we could start the release
> process on Monday, and until then run as many tests as possible.
+1
> Let's make it the best release ever! :)
I have a good feeling about this one. There's some nice marketing
material about crawling efficiency [1]. I should probably extend
benching to indexing and searching too.
[1] http://blog.foofactory.fi/2007/03/twice-speed-half-size.html
--
Sami Siren
Re: Issues pending before 0.9 release
Posted by Andrzej Bialecki <ab...@getopt.org>.
Dennis Kubes wrote
> I did an update, clean, and test and go no errors.
>
> BUILD SUCCESSFUL
> Total time: 6 minutes
I can't figure out what's wrong with the SWF parser when used with JDK
1.6, it works just fine with 1.5 .. However, I propose to add a release
note somewhere that warns about this, and move on with the release anyway.
I upgraded to Hadoop 0.12.2, since it contained some important stability
fixes. All tests pass.
If there are no further issues (anyone?), we could start the release
process on Monday, and until then run as many tests as possible.
Let's make it the best release ever! :)
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Issues pending before 0.9 release
Posted by Dennis Kubes <nu...@dragonflymc.com>.
I did an update, clean, and test and go no errors.
BUILD SUCCESSFUL
Total time: 6 minutes
Sami Siren wrote:
> 2007/3/21, Andrzej Bialecki <ab...@getopt.org>:
>>
>> Sami Siren wrote:
>> > for me it works:
>> >
>> > ...
>> > BUILD SUCCESSFUL
>> > Total time: 4 minutes 3 seconds
>>
>> I did a fresh checkout to an empty dir, rebuilt and it's still failing -
>> perhaps you have some uncommitted changes in your working copy ... ?
>
>
> no, I also did a fresh co from trunk, I'll check it again this evening just
> in case.
>
> --
> Sami Siren
>
Re: Issues pending before 0.9 release
Posted by Sami Siren <ss...@gmail.com>.
2007/3/21, Andrzej Bialecki <ab...@getopt.org>:
>
> Sami Siren wrote:
> > for me it works:
> >
> > ...
> > BUILD SUCCESSFUL
> > Total time: 4 minutes 3 seconds
>
> I did a fresh checkout to an empty dir, rebuilt and it's still failing -
> perhaps you have some uncommitted changes in your working copy ... ?
no, I also did a fresh co from trunk, I'll check it again this evening just
in case.
--
Sami Siren
Re: Issues pending before 0.9 release
Posted by Andrzej Bialecki <ab...@getopt.org>.
Sami Siren wrote:
> for me it works:
>
> ...
> BUILD SUCCESSFUL
> Total time: 4 minutes 3 seconds
I did a fresh checkout to an empty dir, rebuilt and it's still failing -
perhaps you have some uncommitted changes in your working copy ... ?
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Issues pending before 0.9 release
Posted by Sami Siren <ss...@gmail.com>.
for me it works:
...
BUILD SUCCESSFUL
Total time: 4 minutes 3 seconds
--
Sami Siren
2007/3/21, Andrzej Bialecki <ab...@getopt.org>:
>
> Dennis Kubes wrote:
> > I am good to go as well.
>
> Hmm ... Test suite fails for me, with a cryptic message (cryptic because
> the plugin test itself succeeds):
>
> [...]
> init:
>
> init-plugin:
>
> deps-jar:
>
> compile:
> [echo] Compiling plugin: urlnormalizer-regex
>
> compile-test:
>
> jar:
>
> deps-test:
>
> init:
>
> init-plugin:
>
> compile:
>
> jar:
>
> deps-test:
>
> deploy:
>
> copy-generated-lib:
>
> deploy:
>
> copy-generated-lib:
>
> test:
> [echo] Testing plugin: urlnormalizer-regex
> [junit] Running
> org.apache.nutch.net.urlnormalizer.regex.TestRegexURLNormalizer
> [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.016 sec
> [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 8.359 sec
>
> BUILD FAILED
> C:\disks\e\work\nutch\vanilla\build.xml:300: The following error
> occurred while executing this line:
> C:\disks\e\work\nutch\vanilla\src\plugin\build.xml:99: The following
> error occurred while executing this line:
> C:\disks\e\work\nutch\vanilla\src\plugin\build-plugin.xml:200: Tests
> failed!
>
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
Re: Issues pending before 0.9 release
Posted by Andrzej Bialecki <ab...@getopt.org>.
Dennis Kubes wrote:
> I am good to go as well.
Hmm ... Test suite fails for me, with a cryptic message (cryptic because
the plugin test itself succeeds):
[...]
init:
init-plugin:
deps-jar:
compile:
[echo] Compiling plugin: urlnormalizer-regex
compile-test:
jar:
deps-test:
init:
init-plugin:
compile:
jar:
deps-test:
deploy:
copy-generated-lib:
deploy:
copy-generated-lib:
test:
[echo] Testing plugin: urlnormalizer-regex
[junit] Running
org.apache.nutch.net.urlnormalizer.regex.TestRegexURLNormalizer
[junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.016 sec
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 8.359 sec
BUILD FAILED
C:\disks\e\work\nutch\vanilla\build.xml:300: The following error
occurred while executing this line:
C:\disks\e\work\nutch\vanilla\src\plugin\build.xml:99: The following
error occurred while executing this line:
C:\disks\e\work\nutch\vanilla\src\plugin\build-plugin.xml:200: Tests failed!
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Issues pending before 0.9 release
Posted by Dennis Kubes <nu...@dragonflymc.com>.
I am good to go as well.
Dennis Kubes
Andrzej Bialecki wrote:
> Sami Siren wrote:
>> Andrzej Bialecki wrote:
>>> Hi all,
>>>
>>> I just committed Hadoop 0.12.1. Let's double-check that it works ok.
>>> Here's the list of Critical/Blocker issues I mentioned before, and their
>>> current status:
>>>
>>> Any other stuff we need to fix before the release?
>>
>> I am satisfied except the broken bin/nutch.
>
> Fixed now - tested both under Cygwin and Fedora.
>
Re: Issues pending before 0.9 release
Posted by Sami Siren <ss...@gmail.com>.
2007/3/21, Andrzej Bialecki <ab...@getopt.org>:
>
> >> Any other stuff we need to fix before the release?
> >
> > I am satisfied except the broken bin/nutch.
>
> Fixed now - tested both under Cygwin and Fedora.
>
> Thanks, I can confirm that it works now :)
--
Sami Siren
Re: Issues pending before 0.9 release
Posted by Andrzej Bialecki <ab...@getopt.org>.
Sami Siren wrote:
> Andrzej Bialecki wrote:
>> Hi all,
>>
>> I just committed Hadoop 0.12.1. Let's double-check that it works ok.
>> Here's the list of Critical/Blocker issues I mentioned before, and their
>> current status:
>>
>> Any other stuff we need to fix before the release?
>
> I am satisfied except the broken bin/nutch.
Fixed now - tested both under Cygwin and Fedora.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Issues pending before 0.9 release
Posted by Sami Siren <ss...@gmail.com>.
Andrzej Bialecki wrote:
> Hi all,
>
> I just committed Hadoop 0.12.1. Let's double-check that it works ok.
> Here's the list of Critical/Blocker issues I mentioned before, and their
> current status:
>
> Any other stuff we need to fix before the release?
I am satisfied except the broken bin/nutch.
--
Sami Siren
Re: Issues pending before 0.9 release
Posted by Andrzej Bialecki <ab...@getopt.org>.
Hi all,
I just committed Hadoop 0.12.1. Let's double-check that it works ok.
Here's the list of Critical/Blocker issues I mentioned before, and their
current status:
NUTCH-400 Fixed.
NUTCH-353 Moved to Major, fix after release.
NUTCH-233 Fixed.
NUTCH-436 Fixed.
NUTCH-427 Moved to Major, fix after release.
NUTCH-381 Won't fix - this is a configuration issue.
NUTCH-277 Cannot reproduce
NUTCH-167 Fixed.
Any other stuff we need to fix before the release?
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Issues pending before 0.9 release
Posted by Dennis Kubes <nu...@dragonflymc.com>.
> Hi all,
>
> The following issues need to be discussed and appropriate action taken
> before the 0.9 release:
>
> Blocker
> ========
> * NUTCH-400 (Update & add missing license headers) - I believe this is
> fixed and should be closed
>
> * NUTCH-353 (pages that serverside forwards will be refetched every
> time) - this was partially fixed in NUTCH-273, but a more complete
> solution would require significant changes to LinkDb. As there are no
> patches implementing this, I left it open, but it's no longer as
> critical as it was before. I propose to move it to "Major" and address
> it in the next release.
>
> * NUTCH-233 (wrong regular expression hang reduce process for ever) - I
> propose to apply the fix provided by Sean Dean and close this issue for
> now.
>
> Critical
> ========
> * NUTCH-436 (Incorrect handling of relative paths when the embedded URL
> path is empty). There is no patch available yet. If someone could
> contribute a patch I'd like to see this fixed before the release.
I am starting to take a look at this. I will try to get it fixed before
we release.
>
> * NUTCH-427 (protocol-smb). This relies on a LGPL library, and it's
> certainly not critical (as this is an optional new feature). I propose
> to change it to Major, and make a decision - do we want another plugin
> like parse-mp3 or parse-rtf, or not.
>
> * NUTCH-381 (Ignore external link not work as expected) - I'll try to
> reproduce it, and if I find an easy fix I'd like to apply it before the
> release.
>
> * NUTCH-277 (Fetcher dies because of "max. redirects") - I wasn't able
> to reproduce it. If there is no updated information on this I propose to
> close it with "Can't reproduce".
>
> * NUTCH-167 (Observation of <META NAME="ROBOTS" CONTENT="NOARCHIVE">) -
> there's a patch which I tested in a limited production env. If there are
> no objections I'd like to apply it before the release.
>
> Major
> =====
> There are 84 major issues, but some of them are either invalid, or
> should be "minor", or no longer apply and should be closed. Please
> review them if you can and provide some comments or recommendations if
> you think you have some new information.
>
>
> One decision also that we need to make is which version of Hadoop should
> be included in the release. Current trunk uses 0.10.1, I have a set of
> production-tested patches that use 0.11.2, and today the Hadoop team
> released 0.12.0 (to be followed shortly by a 0.12.1, most likely in time
> before our release). The most conservative option is to stay with
> 0.10.1, but by the time people start using Nutch this will be a fairly
> old version already. I propose to upgrade to 0.11.2. We could use 0.12.1
> - but in this case with the expectation that we release less than stable
> version of Nutch to be soon followed by a minor stable release ...
+1 for using 0.11.2. I looked through the release notes for 0.12 and
there were some niceties such as HADOOP-432 for undeletes and alot of bug
fixes, but it didn't look like there were any critical issues as far as
Nutch is concerned.
Dennis Kubes
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
>
Re: Issues pending before 0.9 release
Posted by Sami Siren <ss...@gmail.com>.
Andrzej Bialecki wrote:
> Hi all,
>
> The following issues need to be discussed and appropriate action taken
> before the 0.9 release:
>
> Blocker
> ========
> * NUTCH-400 (Update & add missing license headers) - I believe this is
> fixed and should be closed
I agree. I should close it.
> * NUTCH-233 (wrong regular expression hang reduce process for ever) - I
> propose to apply the fix provided by Sean Dean and close this issue for
> now.
yes that was the resolution also last time :)
> * NUTCH-427 (protocol-smb). This relies on a LGPL library, and it's
> certainly not critical (as this is an optional new feature). I propose
> to change it to Major, and make a decision - do we want another plugin
> like parse-mp3 or parse-rtf, or not.
One option would be setting up a separate project outside Apache to host
and maintain these and remove the remaining torsos from Nutch source base.
> One decision also that we need to make is which version of Hadoop should
> be included in the release. Current trunk uses 0.10.1, I have a set of
> production-tested patches that use 0.11.2, and today the Hadoop team
> released 0.12.0 (to be followed shortly by a 0.12.1, most likely in time
> before our release). The most conservative option is to stay with
> 0.10.1, but by the time people start using Nutch this will be a fairly
0.10.1 is not an option, there is that NPE in sorting that is does not
allow any crawling beyond modes sizes (HADOOP-917). We should upgrade
hadoop to 0.11.2 or 0.12.0 and gather experiences from running it on
reasonable sized crawls, so my suggestion is that don't decide this on
paper.
--
Sami Siren
Re: Issues pending before 0.9 release
Posted by Doug Cutting <cu...@apache.org>.
Sami Siren wrote:
> It would be more beneficial to everybody if the discussions (related to
> release or Nutch) is
> done on public (hey this is open source!). The off the list stuff IMO
> smells.
+1 Folks sometimes wish to discuss project matters off-list to spare
others the boring details, but this is usually a bad idea. All project
decisions should be made in public on this list. Discussions relevant
to these decisions are also thus best made on this list, since they
explain the decision. Private discussions are permissible to develop a
proposal, but that is usually better done on-list when possible, so that
others can get involved earlier.
(The one notable exception is that personnel issues are discussed on the
private PMC list.)
Doug
Re: Issues pending before 0.9 release
Posted by Sami Siren <ss...@gmail.com>.
>
>
>
> P.S. I am going to contact Pitor and coordinate with him: I'd like to be
> the
> release manager for this Nutch release.
>
>
>
It would be more beneficial to everybody if the discussions (related to
release or Nutch) is
done on public (hey this is open source!). The off the list stuff IMO
smells.
--
Sami Siren
Re: Issues pending before 0.9 release
Posted by Dennis Kubes <nu...@dragonflymc.com>.
Chris Mattmann wrote:
> Hi Guys,
>
>> Blocker
>> ========
>> * NUTCH-400 (Update & add missing license headers) - I believe this is
>> fixed and should be closed
>
> +1, thanks to Sami for closing it.
>
>> * NUTCH-353 (pages that serverside forwards will be refetched every
>> time) - this was partially fixed in NUTCH-273, but a more complete
>> solution would require significant changes to LinkDb. As there are no
>> patches implementing this, I left it open, but it's no longer as
>> critical as it was before. I propose to move it to "Major" and address
>> it in the next release.
>
> +1
>
>> * NUTCH-233 (wrong regular expression hang reduce process for ever) - I
>> propose to apply the fix provided by Sean Dean and close this issue for now.
>
> +1
>
>> Critical
>> ========
>> * NUTCH-436 (Incorrect handling of relative paths when the embedded URL
>> path is empty). There is no patch available yet. If someone could
>> contribute a patch I'd like to see this fixed before the release.
>
> Looks like Dennis is on this one
>
>> * NUTCH-427 (protocol-smb). This relies on a LGPL library, and it's
>> certainly not critical (as this is an optional new feature). I propose
>> to change it to Major, and make a decision - do we want another plugin
>> like parse-mp3 or parse-rtf, or not.
>
> Let's hold off on this: it's not necessary for 0.9, and I don't think
> there's been a bunch of traffic on the list identifying this as critical to
> get into the sources for the release
>
>> * NUTCH-381 (Ignore external link not work as expected) - I'll try to
>> reproduce it, and if I find an easy fix I'd like to apply it before the
>> release.
>
> +1
>
>> * NUTCH-277 (Fetcher dies because of "max. redirects") - I wasn't able
>> to reproduce it. If there is no updated information on this I propose to
>> close it with "Can't reproduce".
>
> +1, I had to do something similar with NUTCH-258
>
>> * NUTCH-167 (Observation of <META NAME="ROBOTS" CONTENT="NOARCHIVE">) -
>> there's a patch which I tested in a limited production env. If there are
>> no objections I'd like to apply it before the release.
>
> +1
>
>> Major
>> =====
>> There are 84 major issues, but some of them are either invalid, or
>> should be "minor", or no longer apply and should be closed. Please
>> review them if you can and provide some comments or recommendations if
>> you think you have some new information.
>
> I will spend some time going through JIRA today and see if there's any
> issues that I can find that:
>
> 1. Have a patch already
> 2. Sound like something quick, easy, and not so far-reaching across the
> entire Nutch API
>
>>
>> One decision also that we need to make is which version of Hadoop should
>> be included in the release. Current trunk uses 0.10.1, I have a set of
>> production-tested patches that use 0.11.2, and today the Hadoop team
>> released 0.12.0 (to be followed shortly by a 0.12.1, most likely in time
>> before our release). The most conservative option is to stay with
>> 0.10.1, but by the time people start using Nutch this will be a fairly
>> old version already. I propose to upgrade to 0.11.2. We could use 0.12.1
>> - but in this case with the expectation that we release less than stable
>> version of Nutch to be soon followed by a minor stable release ...
>
> I'd agree with the upgrade to 0.11.2, +1
>
>
> Cheers,
> Chris
>
> P.S. I am going to contact Pitor and coordinate with him: I'd like to be the
> release manager for this Nutch release.
I would like to help with this as well, even if it is just watching how
the process works this time.
Dennis
>
>
>
Re: Issues pending before 0.9 release
Posted by Andrzej Bialecki <ab...@getopt.org>.
Chris Mattmann wrote:
> P.S. I am going to contact Pitor and coordinate with him: I'd like to be the
> release manager for this Nutch release.
>
Everyone heard that? :) That's cool, thanks!
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Issues pending before 0.9 release
Posted by Chris Mattmann <ch...@jpl.nasa.gov>.
Hi Guys,
> Blocker
> ========
> * NUTCH-400 (Update & add missing license headers) - I believe this is
> fixed and should be closed
+1, thanks to Sami for closing it.
>
> * NUTCH-353 (pages that serverside forwards will be refetched every
> time) - this was partially fixed in NUTCH-273, but a more complete
> solution would require significant changes to LinkDb. As there are no
> patches implementing this, I left it open, but it's no longer as
> critical as it was before. I propose to move it to "Major" and address
> it in the next release.
+1
>
> * NUTCH-233 (wrong regular expression hang reduce process for ever) - I
> propose to apply the fix provided by Sean Dean and close this issue for now.
+1
>
> Critical
> ========
> * NUTCH-436 (Incorrect handling of relative paths when the embedded URL
> path is empty). There is no patch available yet. If someone could
> contribute a patch I'd like to see this fixed before the release.
Looks like Dennis is on this one
>
> * NUTCH-427 (protocol-smb). This relies on a LGPL library, and it's
> certainly not critical (as this is an optional new feature). I propose
> to change it to Major, and make a decision - do we want another plugin
> like parse-mp3 or parse-rtf, or not.
Let's hold off on this: it's not necessary for 0.9, and I don't think
there's been a bunch of traffic on the list identifying this as critical to
get into the sources for the release
>
> * NUTCH-381 (Ignore external link not work as expected) - I'll try to
> reproduce it, and if I find an easy fix I'd like to apply it before the
> release.
+1
>
> * NUTCH-277 (Fetcher dies because of "max. redirects") - I wasn't able
> to reproduce it. If there is no updated information on this I propose to
> close it with "Can't reproduce".
+1, I had to do something similar with NUTCH-258
>
> * NUTCH-167 (Observation of <META NAME="ROBOTS" CONTENT="NOARCHIVE">) -
> there's a patch which I tested in a limited production env. If there are
> no objections I'd like to apply it before the release.
+1
>
> Major
> =====
> There are 84 major issues, but some of them are either invalid, or
> should be "minor", or no longer apply and should be closed. Please
> review them if you can and provide some comments or recommendations if
> you think you have some new information.
I will spend some time going through JIRA today and see if there's any
issues that I can find that:
1. Have a patch already
2. Sound like something quick, easy, and not so far-reaching across the
entire Nutch API
>
>
> One decision also that we need to make is which version of Hadoop should
> be included in the release. Current trunk uses 0.10.1, I have a set of
> production-tested patches that use 0.11.2, and today the Hadoop team
> released 0.12.0 (to be followed shortly by a 0.12.1, most likely in time
> before our release). The most conservative option is to stay with
> 0.10.1, but by the time people start using Nutch this will be a fairly
> old version already. I propose to upgrade to 0.11.2. We could use 0.12.1
> - but in this case with the expectation that we release less than stable
> version of Nutch to be soon followed by a minor stable release ...
I'd agree with the upgrade to 0.11.2, +1
Cheers,
Chris
P.S. I am going to contact Pitor and coordinate with him: I'd like to be the
release manager for this Nutch release.