You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> on 2010/06/26 20:26:04 UTC
Nutch 2.0
Hi All,
We just wanted to flush some our discussions on Nutch 2.0 to the public
mailing lists. A few key points:
1. Doğacan, Julien and Enis are proposing to base Nutch 2.0 on GORA, a
technology that can be viewed here [1].
The upshot of GORA is that it's an ORM layer that abstracts away backend
stores like Cassandra or SQL or any other database (though Gora works better
with NoSQL stores).
A couple of questions I've got:
a) is GORA ASL licensed?
b) what's the maintenance plan for GORA? Will it continue to live in
Github? Will you guys propose it into the Apache Incubator as an ASF
project?
2. Development on Nutchbase has occurred at Github [2] since Doğacan
originally checked it into Nutch SVN under a branch [3] at the ASF. I
expressed some concerns about this since it's hard to review huge sweeping
patches and since a lot of development has occurred off of the public Apache
mailing lists. Specifically I asked Doğacan et al. to enumerate a list of
the changes in the Git version of Nutchbase [2] versus the ASF version [3].
We then need to come up with a plan of how to merge the 2 and get the latest
into ASF SVN. Doğacan estimates the difference between the Nutchbase branch
at the ASF [3] and that of Github [2] to be ~25 hrs of work. Doğacan
generated this list of major changes that have happened at Github and not at
Apache:
---snip
1) Porting nutchbase to GORA: This was discussed in issues NUTCH-808
and NUTCH-811.
2) Using ivy in nutch: NUTCH-821 and NUTCH-825
3) Removal of nutch's custom developed search code (and using SOLR instead).
IIRC, this was also
discussed and accepted by nutch community. However, if not, we can simply
put this code back (since
this was a trivial delete).
---snip
So, that really brings everyone up to speed I think. So, that said, I am +1
for moving forward on #1 above, provided we address the 2 questions I listed
(a+b). We need to understand it from a Nutch perspective. As for #2, we can
rectify it by doing the following things:
(a) svn copy NutchBase from GitHub to the nutchbase branch in
http://svn.apache.org/repos/asf/nutch/branches/nutchbase bringing the ASF
branch up to date.
(b) Once the GORA license issues are figured out (they must be
compatible with the ASF or we cannot use it), then we update Nutch to depend
on the GORA jars via Ivy?
(c) svn tag current Nutch trunk as 1.2-branch
(d) svn merge nutchbase branch with nutch trunk
(e) roll the version # in nutch trunk to 2.0-dev
(f) all issues in JIRA should be updated to reflect 2.0-dev fixes where
it makes sense
(g) a 2.1 version is added to mark anything that we don't want in 2.0
and we file post 2.0 issues there
(h) Nutch 2.0 trunk is fixed, and brought up to speed and old code is
removed. All unit tests should pass regression where it makes sense.
(i) Nutch documentation is brought up to date on wiki and checked into
SVN
(j) We roll a 2.0 release
That sound good to everyone?
Cheers,
Chris
[1] http://github.com/enis/gora
[2] http://github.com/dogacan/nutchbase
[3] http://svn.apache.org/repos/asf/nutch/branches/nutchbase/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Nutch 2.0
Posted by Doğacan Güney <do...@gmail.com>.
Hi all,
On Sat, Jun 26, 2010 at 21:26, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi All,
>
> We just wanted to flush some our discussions on Nutch 2.0 to the public
> mailing lists. A few key points:
>
> 1. Doğacan, Julien and Enis are proposing to base Nutch 2.0 on GORA, a
> technology that can be viewed here [1].
>
> The upshot of GORA is that it's an ORM layer that abstracts away backend
> stores like Cassandra or SQL or any other database (though Gora works
> better
> with NoSQL stores).
>
> A couple of questions I've got:
>
> a) is GORA ASL licensed?
>
I think it is already ASL. Even if it isn't (which means we forgot to do
so),
we have no problems about making it ASL-licensed.
> b) what's the maintenance plan for GORA? Will it continue to live in
> Github? Will you guys propose it into the Apache Incubator as an ASF
> project?
>
>
That all depends on the amount of interest GORA generates. If there is
interest, we will
be more than happy to propose it as an incubator project.
> 2. Development on Nutchbase has occurred at Github [2] since Doğacan
> originally checked it into Nutch SVN under a branch [3] at the ASF. I
> expressed some concerns about this since it's hard to review huge sweeping
> patches and since a lot of development has occurred off of the public
> Apache
> mailing lists. Specifically I asked Doğacan et al. to enumerate a list of
> the changes in the Git version of Nutchbase [2] versus the ASF version [3].
> We then need to come up with a plan of how to merge the 2 and get the
> latest
> into ASF SVN. Doğacan estimates the difference between the Nutchbase branch
> at the ASF [3] and that of Github [2] to be ~25 hrs of work. Doğacan
> generated this list of major changes that have happened at Github and not
> at
> Apache:
>
---snip
> 1) Porting nutchbase to GORA: This was discussed in issues NUTCH-808
> and NUTCH-811.
> 2) Using ivy in nutch: NUTCH-821 and NUTCH-825
> 3) Removal of nutch's custom developed search code (and using SOLR
> instead).
> IIRC, this was also
> discussed and accepted by nutch community. However, if not, we can simply
> put this code back (since
> this was a trivial delete).
> ---snip
>
> So, that really brings everyone up to speed I think. So, that said, I am +1
> for moving forward on #1 above, provided we address the 2 questions I
> listed
> (a+b). We need to understand it from a Nutch perspective. As for #2, we can
> rectify it by doing the following things:
>
> (a) svn copy NutchBase from GitHub to the nutchbase branch in
> http://svn.apache.org/repos/asf/nutch/branches/nutchbase bringing the ASF
> branch up to date.
> (b) Once the GORA license issues are figured out (they must be
> compatible with the ASF or we cannot use it), then we update Nutch to
> depend
> on the GORA jars via Ivy?
> (c) svn tag current Nutch trunk as 1.2-branch
> (d) svn merge nutchbase branch with nutch trunk
> (e) roll the version # in nutch trunk to 2.0-dev
> (f) all issues in JIRA should be updated to reflect 2.0-dev fixes where
> it makes sense
> (g) a 2.1 version is added to mark anything that we don't want in 2.0
> and we file post 2.0 issues there
> (h) Nutch 2.0 trunk is fixed, and brought up to speed and old code is
> removed. All unit tests should pass regression where it makes sense.
> (i) Nutch documentation is brought up to date on wiki and checked into
> SVN
> (j) We roll a 2.0 release
>
> That sound good to everyone?
>
>
Sounds good.
> Cheers,
> Chris
>
> [1] http://github.com/enis/gora
> [2] http://github.com/dogacan/nutchbase
> [3] http://svn.apache.org/repos/asf/nutch/branches/nutchbase/
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: Chris.Mattmann@jpl.nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
--
Doğacan Güney
Re: Nutch 2.0
Posted by Julien Nioche <li...@gmail.com>.
Hi Dogacan
I will double check to make sure, but IIRC, there is no need to delete
> svn:nutchbase since current code on
> github simply builds on top of that.
>
So why not simply merge github branch into svn? It will be a clear merge...
>
>
The only problem is contributor info is messed up in github but I tried to
> preserve as much contrib info as possible
> when I pulled in 1.1 changes (via git cherry-pick). So we can break the
> code in github into smaller patches, apply them
> on top of svn nutchbase (which, again, will be clean) then, 1.1 changes can
> be applied by _original_ committers, thus
>
hopefully preserving contributor info as well.
>
I thought you had massively rewritten nutchbase in github but if we can
merge cleanly that would be even better :-)
We'll still need to patch the trunk from svn nutchbase after that anyway and
the 1.1 changes are already there so I suppose the info about the
contributors is not vital. Or am I missing something?
>
> Makes sense?
>
> On Mon, Jun 28, 2010 at 16:45, Julien Nioche <
> lists.digitalpebble@gmail.com> wrote:
>
>> Hi,
>>
>> (a) deleting svn:nutchbase
>>> (b) svn:importing Git Nutchbase.
>>> (c) branch current 1.2-trunk as 1.2-branch
>>> (d) iteratively apply patches from new svn:nutchbase to trunk to
>>> bring
>>> it up to snuff.
>>> (e) roll the version # in nutch trunk to 2.0-dev
>>> (f) all issues in JIRA should be updated to reflect 2.0-dev fixes
>>> where
>>> it makes sense
>>> (g) a 2.1 version is added to mark anything that we don't want in 2.0
>>> and we file post 2.0 issues there
>>> (h) Nutch 2.0 trunk is fixed, and brought up to speed and old code is
>>> removed. All unit tests should pass regression where it makes sense.
>>> (i) Nutch documentation is brought up to date on wiki and checked
>>> into
>>> SVN
>>> (j) We roll a 2.0 release
>>>
>>
>> +1
>>
>>
>>>
>>> I'd be happy to do (a), (c), (e) and (g) tomorrow, and would like to
>>> participate in (d) and (f).
>>
>> I'm thinking Julien and Doğacan would be the
>>> best people to do (b) and (i).
>>
>>
>> Doğacan is in the process of writing the documentation
>>
>>
>>
>>> (h) should be a result of all steps prior
>>> (a)-(g), and as for (j), I'd be happy to do (j) when the time comes.
>>>
>>> So, if I don't hear any objections, I'll do (a), (c), (e) and (g)
>>> tomorrow... (6/28, likely PM PST Los Angeles time)
>>
>>
>> cool, thanks
>>
>> J.
>> --
>> DigitalPebble Ltd
>>
>> Open Source Solutions for Text Engineering
>> http://www.digitalpebble.com
>>
>
>
>
> --
> Doğacan Güney
>
>
--
DigitalPebble Ltd
Open Source Solutions for Text Engineering
http://www.digitalpebble.com
Re: Nutch 2.0
Posted by Julien Nioche <li...@gmail.com>.
2010/6/29 Do�acan G�ney <do...@gmail.com>
> Hi,
>
> On Tue, Jun 29, 2010 at 11:49, Julien Nioche <
> lists.digitalpebble@gmail.com> wrote:
>
>> Thanks Chris,
>>
>> I already shared my thoughts on this yesterday, but I still fail to see
>> the advantage of keeping the details of the recent github nutchbase commits
>> (some of them being just upgrades to the recent changes in 1.1) in svn
>> nutchbase knowing that the point is actually to do incremental changes to
>> the existing trunk (which already has the 1.1 changes) from svn nutchbase
>> and review / comment / improve the code on this occasion.
>>
>> Since we also want to produce a patch in JIRA for the changes in svn
>> nutchbase in order to put the "donated to Apache" stamp on it it would make
>> sense to do that just once and not for all the commits which have been done
>> in github.
>>
>> I am probably missing an important point here, but if so I would
>> appreciate if someone (Dogacan?) could explain why we should not stick to
>> the original plan
>> (a) clear the existing svn nutchbase
>> (b) generate a large patch with the code from github and JIRA it
>
>
> Do you mean generating a single patch vs nutch? There are a lot of fixes
> and improvements in nutch 1.1 that I cherry-picked to nutchbase later. If we
> generate
> a larger patch, and then this branch is blessed as "trunk" then history for
> those improvements will be lost.
>
> Or am I misunderstanding you here?
>
as explained earlier we want *github nutchbase* to be transfered integrally
to* svn nutchbase*. Then we want to progressively patch the *trunk*
from*svn nutchbase
*. Since the trunk is 1.1 all its features are already there as well as
their history. we are not interested in the history of the changes within
nutchbase (be it then svn or github one). The consensus was that it would be
easier to delete svn nutchbase then port the content of github nutchbase in
one go. The single patch on JIRA would be used not so much to review or
track the modifications but to clear the issues around licensing and code
donation.
makes sense?
>
>
>>
>
> (c) commit the changes to svn nutchbase
>> then get on with the interesting bits.
>>
>> My concern is that proceeding as Dogacan described yesterday might take
>> quite some time and block the rest of the work on 2.0. I am happy to work on
>> the 3 steps above BTW.
>>
>> Thanks
>>
>> Julien
>>
>>
>>
>>
>>
>> On 29 June 2010 06:44, Mattmann, Chris A (388J) <
>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>
>>> Okey dokey guys, (c), (e) and (g) are done.
>>>
>>> Julien, Do�acan, your turn on (a) and (d) and then we can all work on (e)
>>> and (f)...
>>>
>>> Cheers,
>>> Chris
>>>
>>>
>>>
>>>
>>> On 6/28/10 12:55 PM, "Do�acan G�ney" <do...@gmail.com> wrote:
>>>
>>> On Mon, Jun 28, 2010 at 20:23, Andrzej Bialecki <ab...@getopt.org> wrote:
>>>
>>> On 2010-06-28 17:57, Mattmann, Chris A (388J) wrote:
>>> > Hi Do�acan,
>>> >
>>> > So your proposition is to combine (a) and (b) then? That's fine by me,
>>> > so long as there are no objections from others. I can still move
>>> forward
>>> > with , (e) and (g) then...
>>>
>>>
>>> No objections from me - but IMHO to satisfy the legal minds you still
>>> need to produce a patch and attach to an issue with the "Grant to ASF"
>>> checkbox marked...
>>>
>>>
>>> OK, I'll create a new issue in JIRA, and then attach a lot of patches :)
>>>
>>> I'll try to appropriately mark patches that are straightforward ports
>>> from nutch 1.1
>>> into nutchbase so that the same committers can commit those patches
>>> _again_
>>> hopefully preserving post nutch 1.0 history as much as possible.
>>>
>>>
>>> (Also, I always shudder when I imagine a massive merge failing ... but
>>> that's probably a leftover from my CVS days when a failed merge would
>>> leave a completely broken tree.. ah, well, good luck :) ).
>>>
>>>
>>> I regularly do large merges in git and it works beautifully. We'll see
>>> how well
>>> SVN does :)
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Andrzej Bialecki <><
>>> ___. ___ ___ ___ _ _ __________________________________
>>> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
>>> ___|||__|| \| || | Embedded Unix, System Integration
>>> http://www.sigram.com Contact: info at sigram dot com
>>>
>>>
>>>
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: *Chris.Mattmann@jpl.nasa.gov
>>> *WWW: *http://sunset.usc.edu/~mattmann/<http://sunset.usc.edu/%7Emattmann/>
>>> *++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>>
>>
>>
>> --
>> DigitalPebble Ltd
>>
>> Open Source Solutions for Text Engineering
>> http://www.digitalpebble.com
>>
>
>
>
> --
> Do�acan G�ney
>
>
--
DigitalPebble Ltd
Open Source Solutions for Text Engineering
http://www.digitalpebble.com
Re: Nutch 2.0
Posted by Doğacan Güney <do...@gmail.com>.
Hi,
On Tue, Jun 29, 2010 at 11:49, Julien Nioche
<li...@gmail.com>wrote:
> Thanks Chris,
>
> I already shared my thoughts on this yesterday, but I still fail to see the
> advantage of keeping the details of the recent github nutchbase commits
> (some of them being just upgrades to the recent changes in 1.1) in svn
> nutchbase knowing that the point is actually to do incremental changes to
> the existing trunk (which already has the 1.1 changes) from svn nutchbase
> and review / comment / improve the code on this occasion.
>
> Since we also want to produce a patch in JIRA for the changes in svn
> nutchbase in order to put the "donated to Apache" stamp on it it would make
> sense to do that just once and not for all the commits which have been done
> in github.
>
> I am probably missing an important point here, but if so I would appreciate
> if someone (Dogacan?) could explain why we should not stick to the original
> plan
> (a) clear the existing svn nutchbase
> (b) generate a large patch with the code from github and JIRA it
Do you mean generating a single patch vs nutch? There are a lot of fixes and
improvements in nutch 1.1 that I cherry-picked to nutchbase later. If we
generate
a larger patch, and then this branch is blessed as "trunk" then history for
those improvements will be lost.
Or am I misunderstanding you here?
>
(c) commit the changes to svn nutchbase
> then get on with the interesting bits.
>
> My concern is that proceeding as Dogacan described yesterday might take
> quite some time and block the rest of the work on 2.0. I am happy to work on
> the 3 steps above BTW.
>
> Thanks
>
> Julien
>
>
>
>
>
> On 29 June 2010 06:44, Mattmann, Chris A (388J) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Okey dokey guys, (c), (e) and (g) are done.
>>
>> Julien, Doğacan, your turn on (a) and (d) and then we can all work on (e)
>> and (f)...
>>
>> Cheers,
>> Chris
>>
>>
>>
>>
>> On 6/28/10 12:55 PM, "Doğacan Güney" <do...@gmail.com> wrote:
>>
>> On Mon, Jun 28, 2010 at 20:23, Andrzej Bialecki <ab...@getopt.org> wrote:
>>
>> On 2010-06-28 17:57, Mattmann, Chris A (388J) wrote:
>> > Hi Doğacan,
>> >
>> > So your proposition is to combine (a) and (b) then? That’s fine by me,
>> > so long as there are no objections from others. I can still move forward
>> > with , (e) and (g) then...
>>
>>
>> No objections from me - but IMHO to satisfy the legal minds you still
>> need to produce a patch and attach to an issue with the "Grant to ASF"
>> checkbox marked...
>>
>>
>> OK, I'll create a new issue in JIRA, and then attach a lot of patches :)
>>
>> I'll try to appropriately mark patches that are straightforward ports from
>> nutch 1.1
>> into nutchbase so that the same committers can commit those patches
>> _again_
>> hopefully preserving post nutch 1.0 history as much as possible.
>>
>>
>> (Also, I always shudder when I imagine a massive merge failing ... but
>> that's probably a leftover from my CVS days when a failed merge would
>> leave a completely broken tree.. ah, well, good luck :) ).
>>
>>
>> I regularly do large merges in git and it works beautifully. We'll see how
>> well
>> SVN does :)
>>
>>
>>
>> --
>> Best regards,
>> Andrzej Bialecki <><
>> ___. ___ ___ ___ _ _ __________________________________
>> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
>> ___|||__|| \| || | Embedded Unix, System Integration
>> http://www.sigram.com Contact: info at sigram dot com
>>
>>
>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: *Chris.Mattmann@jpl.nasa.gov
>> *WWW: *http://sunset.usc.edu/~mattmann/<http://sunset.usc.edu/%7Emattmann/>
>> *++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>
>
> --
> DigitalPebble Ltd
>
> Open Source Solutions for Text Engineering
> http://www.digitalpebble.com
>
--
Doğacan Güney
Re: Nutch 2.0
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Guys,
On 6/29/10 2:30 AM, "Andrzej Bialecki" <ab...@getopt.org> wrote:
>> I am probably missing an important point here, but if so I would
>> appreciate if someone (Dogacan?) could explain why we should not stick
>> to the original plan
>> (a) clear the existing svn nutchbase
>> (b) generate a large patch with the code from github and JIRA it
>> (c) commit the changes to svn nutchbase
>> then get on with the interesting bits.
Like I said, whether we merge the Github Nutchbase into the Apache Nutchbase
branch or we blow away the Apache Nutchbase branch and then import the
Github Nutchbase branch wholesale, either way, we are left with an Apache
Nutchbase branch that needs to incrementally be merged into the Nutch 2.0
trunk, which I agree with Andrzej, and Julien, is the most important part.
So, either way works fine with me, so long as we are left with an Apache
Nutchbase branch that can be merged incrementally with the Apache Nutch 2.0
trunk. I'm just not going to be the one doing that first part (Github
transfer), so I didn't want to push one way or another.
Once the Apache Nutchbase branch is ready, can we identify a set of 5-10
JIRA patches that we can use to track how to bring the Apache Nutchbase
branch into the Apache Nutch 2.0 trunk? At that point, I'll likely be of use
again :) Until then, Julien, Dogacan, I think the floor is yours.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Nutch 2.0
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-06-29 10:49, Julien Nioche wrote:
> Thanks Chris,
>
> I already shared my thoughts on this yesterday, but I still fail to see
> the advantage of keeping the details of the recent github nutchbase
> commits (some of them being just upgrades to the recent changes in 1.1)
> in svn nutchbase knowing that the point is actually to do incremental
> changes to the existing trunk (which already has the 1.1 changes) from
> svn nutchbase and review / comment / improve the code on this occasion.
I agree with Julien - given that we are going to merge the code
piecewise I have no use for the github history. At this point the github
code is just a series of patches, and the way we merge these patches
with trunk is what creates a relevant history IMHO.
> I am probably missing an important point here, but if so I would
> appreciate if someone (Dogacan?) could explain why we should not stick
> to the original plan
> (a) clear the existing svn nutchbase
> (b) generate a large patch with the code from github and JIRA it
> (c) commit the changes to svn nutchbase
> then get on with the interesting bits.
+1.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Nutch 2.0
Posted by Julien Nioche <li...@gmail.com>.
Thanks Chris,
I already shared my thoughts on this yesterday, but I still fail to see the
advantage of keeping the details of the recent github nutchbase commits
(some of them being just upgrades to the recent changes in 1.1) in svn
nutchbase knowing that the point is actually to do incremental changes to
the existing trunk (which already has the 1.1 changes) from svn nutchbase
and review / comment / improve the code on this occasion.
Since we also want to produce a patch in JIRA for the changes in svn
nutchbase in order to put the "donated to Apache" stamp on it it would make
sense to do that just once and not for all the commits which have been done
in github.
I am probably missing an important point here, but if so I would appreciate
if someone (Dogacan?) could explain why we should not stick to the original
plan
(a) clear the existing svn nutchbase
(b) generate a large patch with the code from github and JIRA it
(c) commit the changes to svn nutchbase
then get on with the interesting bits.
My concern is that proceeding as Dogacan described yesterday might take
quite some time and block the rest of the work on 2.0. I am happy to work on
the 3 steps above BTW.
Thanks
Julien
On 29 June 2010 06:44, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Okey dokey guys, (c), (e) and (g) are done.
>
> Julien, Doğacan, your turn on (a) and (d) and then we can all work on (e)
> and (f)...
>
> Cheers,
> Chris
>
>
>
>
> On 6/28/10 12:55 PM, "Doğacan Güney" <do...@gmail.com> wrote:
>
> On Mon, Jun 28, 2010 at 20:23, Andrzej Bialecki <ab...@getopt.org> wrote:
>
> On 2010-06-28 17:57, Mattmann, Chris A (388J) wrote:
> > Hi Doğacan,
> >
> > So your proposition is to combine (a) and (b) then? That’s fine by me,
> > so long as there are no objections from others. I can still move forward
> > with , (e) and (g) then...
>
>
> No objections from me - but IMHO to satisfy the legal minds you still
> need to produce a patch and attach to an issue with the "Grant to ASF"
> checkbox marked...
>
>
> OK, I'll create a new issue in JIRA, and then attach a lot of patches :)
>
> I'll try to appropriately mark patches that are straightforward ports from
> nutch 1.1
> into nutchbase so that the same committers can commit those patches _again_
> hopefully preserving post nutch 1.0 history as much as possible.
>
>
> (Also, I always shudder when I imagine a massive merge failing ... but
> that's probably a leftover from my CVS days when a failed merge would
> leave a completely broken tree.. ah, well, good luck :) ).
>
>
> I regularly do large merges in git and it works beautifully. We'll see how
> well
> SVN does :)
>
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: *Chris.Mattmann@jpl.nasa.gov
> *WWW: *http://sunset.usc.edu/~mattmann/<http://sunset.usc.edu/%7Emattmann/>
> *++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
--
DigitalPebble Ltd
Open Source Solutions for Text Engineering
http://www.digitalpebble.com
Re: Nutch 2.0
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Okey dokey guys, (c), (e) and (g) are done.
Julien, Doğacan, your turn on (a) and (d) and then we can all work on (e) and (f)...
Cheers,
Chris
On 6/28/10 12:55 PM, "Doğacan Güney" <do...@gmail.com> wrote:
On Mon, Jun 28, 2010 at 20:23, Andrzej Bialecki <ab...@getopt.org> wrote:
On 2010-06-28 17:57, Mattmann, Chris A (388J) wrote:
> Hi Doğacan,
>
> So your proposition is to combine (a) and (b) then? That’s fine by me,
> so long as there are no objections from others. I can still move forward
> with , (e) and (g) then...
No objections from me - but IMHO to satisfy the legal minds you still
need to produce a patch and attach to an issue with the "Grant to ASF"
checkbox marked...
OK, I'll create a new issue in JIRA, and then attach a lot of patches :)
I'll try to appropriately mark patches that are straightforward ports from nutch 1.1
into nutchbase so that the same committers can commit those patches _again_
hopefully preserving post nutch 1.0 history as much as possible.
(Also, I always shudder when I imagine a massive merge failing ... but
that's probably a leftover from my CVS days when a failed merge would
leave a completely broken tree.. ah, well, good luck :) ).
I regularly do large merges in git and it works beautifully. We'll see how well
SVN does :)
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Nutch 2.0
Posted by Doğacan Güney <do...@gmail.com>.
On Mon, Jun 28, 2010 at 20:23, Andrzej Bialecki <ab...@getopt.org> wrote:
> On 2010-06-28 17:57, Mattmann, Chris A (388J) wrote:
> > Hi Doğacan,
> >
> > So your proposition is to combine (a) and (b) then? That’s fine by me,
> > so long as there are no objections from others. I can still move forward
> > with , (e) and (g) then...
>
>
> No objections from me - but IMHO to satisfy the legal minds you still
> need to produce a patch and attach to an issue with the "Grant to ASF"
> checkbox marked...
>
>
OK, I'll create a new issue in JIRA, and then attach a lot of patches :)
I'll try to appropriately mark patches that are straightforward ports from
nutch 1.1
into nutchbase so that the same committers can commit those patches _again_
hopefully preserving post nutch 1.0 history as much as possible.
> (Also, I always shudder when I imagine a massive merge failing ... but
> that's probably a leftover from my CVS days when a failed merge would
> leave a completely broken tree.. ah, well, good luck :) ).
>
>
I regularly do large merges in git and it works beautifully. We'll see how
well
SVN does :)
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
--
Doğacan Güney
Re: Nutch 2.0
Posted by Kirby Bohling <ki...@gmail.com>.
On Mon, Jun 28, 2010 at 12:23 PM, Andrzej Bialecki <ab...@getopt.org> wrote:
> On 2010-06-28 17:57, Mattmann, Chris A (388J) wrote:
>> Hi Doğacan,
>>
<snip...>
> (Also, I always shudder when I imagine a massive merge failing ... but
> that's probably a leftover from my CVS days when a failed merge would
> leave a completely broken tree.. ah, well, good luck :) ).
As someone who has successfully used the "git svn" functionality daily
at work to use git in my day to day work, but satisfy my corporate
bosses by using SVN, I'd highly recommend trying that out.
git.apache.org should make it pretty easy to get the github code and
the SVN history into a single repository. Then allow the use of much
more sophisticated git tools to re-build a history that satisfies
everyone, and then pull the patches and stuff them into JIRA.
I maintain all of my out-of-tree Nutch work that way. Not being a
committer for Apache, I don't know how well the svn dcommit stuff
works with the git exports and the official Apache SVN server. Works
great at the office.
Kirby
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
Re: Nutch 2.0
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-06-28 17:57, Mattmann, Chris A (388J) wrote:
> Hi Doğacan,
>
> So your proposition is to combine (a) and (b) then? That’s fine by me,
> so long as there are no objections from others. I can still move forward
> with , (e) and (g) then...
No objections from me - but IMHO to satisfy the legal minds you still
need to produce a patch and attach to an issue with the "Grant to ASF"
checkbox marked...
(Also, I always shudder when I imagine a massive merge failing ... but
that's probably a leftover from my CVS days when a failed merge would
leave a completely broken tree.. ah, well, good luck :) ).
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Nutch 2.0
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Guys,
And, let me clarify my OK’ness with this. My assumption is that regardless of whether we physically svn:delete nutchbase in Apache SVN (the choice I went to after hearing there were *significant* changes in the Git version from that of the Apache one), and then import a fresh copy from Git, or whether we simply update Nutchbase in apache SVN with Git patches (my original suggestion), that in the end, we are left with a Nutchbase branch that we can move forward from in Apache SVN.
If that is the case, then I think my suggested plan below applies either way and we can move forward...
Cheers,
Chris
On 6/28/10 8:57 AM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
Hi Doğacan,
So your proposition is to combine (a) and (b) then? That’s fine by me, so long as there are no objections from others. I can still move forward with , (e) and (g) then...
Cheers,
Chris
On 6/28/10 8:39 AM, "Doğacan Güney" <do...@gmail.com> wrote:
Hey all,
I will double check to make sure, but IIRC, there is no need to delete svn:nutchbase since current code on
github simply builds on top of that. So why not simply merge github branch into svn? It will be a clear merge...
The only problem is contributor info is messed up in github but I tried to preserve as much contrib info as possible
when I pulled in 1.1 changes (via git cherry-pick). So we can break the code in github into smaller patches, apply them
on top of svn nutchbase (which, again, will be clean) then, 1.1 changes can be applied by _original_ committers, thus
hopefully preserving contributor info as well.
Makes sense?
On Mon, Jun 28, 2010 at 16:45, Julien Nioche <li...@gmail.com> wrote:
Hi,
(a) deleting svn:nutchbase
(b) svn:importing Git Nutchbase.
(c) branch current 1.2-trunk as 1.2-branch
(d) iteratively apply patches from new svn:nutchbase to trunk to bring
it up to snuff.
(e) roll the version # in nutch trunk to 2.0-dev
(f) all issues in JIRA should be updated to reflect 2.0-dev fixes where
it makes sense
(g) a 2.1 version is added to mark anything that we don't want in 2.0
and we file post 2.0 issues there
(h) Nutch 2.0 trunk is fixed, and brought up to speed and old code is
removed. All unit tests should pass regression where it makes sense.
(i) Nutch documentation is brought up to date on wiki and checked into
SVN
(j) We roll a 2.0 release
+1
I'd be happy to do (a), (c), (e) and (g) tomorrow, and would like to
participate in (d) and (f).
I'm thinking Julien and Doğacan would be the
best people to do (b) and (i).
Doğacan is in the process of writing the documentation
(h) should be a result of all steps prior
(a)-(g), and as for (j), I'd be happy to do (j) when the time comes.
So, if I don't hear any objections, I'll do (a), (c), (e) and (g)
tomorrow... (6/28, likely PM PST Los Angeles time)
cool, thanks
J.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Nutch 2.0
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Doğacan,
So your proposition is to combine (a) and (b) then? That’s fine by me, so long as there are no objections from others. I can still move forward with , (e) and (g) then...
Cheers,
Chris
On 6/28/10 8:39 AM, "Doğacan Güney" <do...@gmail.com> wrote:
Hey all,
I will double check to make sure, but IIRC, there is no need to delete svn:nutchbase since current code on
github simply builds on top of that. So why not simply merge github branch into svn? It will be a clear merge...
The only problem is contributor info is messed up in github but I tried to preserve as much contrib info as possible
when I pulled in 1.1 changes (via git cherry-pick). So we can break the code in github into smaller patches, apply them
on top of svn nutchbase (which, again, will be clean) then, 1.1 changes can be applied by _original_ committers, thus
hopefully preserving contributor info as well.
Makes sense?
On Mon, Jun 28, 2010 at 16:45, Julien Nioche <li...@gmail.com> wrote:
Hi,
(a) deleting svn:nutchbase
(b) svn:importing Git Nutchbase.
(c) branch current 1.2-trunk as 1.2-branch
(d) iteratively apply patches from new svn:nutchbase to trunk to bring
it up to snuff.
(e) roll the version # in nutch trunk to 2.0-dev
(f) all issues in JIRA should be updated to reflect 2.0-dev fixes where
it makes sense
(g) a 2.1 version is added to mark anything that we don't want in 2.0
and we file post 2.0 issues there
(h) Nutch 2.0 trunk is fixed, and brought up to speed and old code is
removed. All unit tests should pass regression where it makes sense.
(i) Nutch documentation is brought up to date on wiki and checked into
SVN
(j) We roll a 2.0 release
+1
I'd be happy to do (a), (c), (e) and (g) tomorrow, and would like to
participate in (d) and (f).
I'm thinking Julien and Doğacan would be the
best people to do (b) and (i).
Doğacan is in the process of writing the documentation
(h) should be a result of all steps prior
(a)-(g), and as for (j), I'd be happy to do (j) when the time comes.
So, if I don't hear any objections, I'll do (a), (c), (e) and (g)
tomorrow... (6/28, likely PM PST Los Angeles time)
cool, thanks
J.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Nutch 2.0
Posted by Doğacan Güney <do...@gmail.com>.
Hey all,
I will double check to make sure, but IIRC, there is no need to delete
svn:nutchbase since current code on
github simply builds on top of that. So why not simply merge github branch
into svn? It will be a clear merge...
The only problem is contributor info is messed up in github but I tried to
preserve as much contrib info as possible
when I pulled in 1.1 changes (via git cherry-pick). So we can break the code
in github into smaller patches, apply them
on top of svn nutchbase (which, again, will be clean) then, 1.1 changes can
be applied by _original_ committers, thus
hopefully preserving contributor info as well.
Makes sense?
On Mon, Jun 28, 2010 at 16:45, Julien Nioche
<li...@gmail.com>wrote:
> Hi,
>
> (a) deleting svn:nutchbase
>> (b) svn:importing Git Nutchbase.
>> (c) branch current 1.2-trunk as 1.2-branch
>> (d) iteratively apply patches from new svn:nutchbase to trunk to bring
>> it up to snuff.
>> (e) roll the version # in nutch trunk to 2.0-dev
>> (f) all issues in JIRA should be updated to reflect 2.0-dev fixes
>> where
>> it makes sense
>> (g) a 2.1 version is added to mark anything that we don't want in 2.0
>> and we file post 2.0 issues there
>> (h) Nutch 2.0 trunk is fixed, and brought up to speed and old code is
>> removed. All unit tests should pass regression where it makes sense.
>> (i) Nutch documentation is brought up to date on wiki and checked into
>> SVN
>> (j) We roll a 2.0 release
>>
>
> +1
>
>
>>
>> I'd be happy to do (a), (c), (e) and (g) tomorrow, and would like to
>> participate in (d) and (f).
>
> I'm thinking Julien and Doğacan would be the
>> best people to do (b) and (i).
>
>
> Doğacan is in the process of writing the documentation
>
>
>
>> (h) should be a result of all steps prior
>> (a)-(g), and as for (j), I'd be happy to do (j) when the time comes.
>>
>> So, if I don't hear any objections, I'll do (a), (c), (e) and (g)
>> tomorrow... (6/28, likely PM PST Los Angeles time)
>
>
> cool, thanks
>
> J.
> --
> DigitalPebble Ltd
>
> Open Source Solutions for Text Engineering
> http://www.digitalpebble.com
>
--
Doğacan Güney
Re: Nutch 2.0
Posted by Julien Nioche <li...@gmail.com>.
Hi,
(a) deleting svn:nutchbase
> (b) svn:importing Git Nutchbase.
> (c) branch current 1.2-trunk as 1.2-branch
> (d) iteratively apply patches from new svn:nutchbase to trunk to bring
> it up to snuff.
> (e) roll the version # in nutch trunk to 2.0-dev
> (f) all issues in JIRA should be updated to reflect 2.0-dev fixes where
> it makes sense
> (g) a 2.1 version is added to mark anything that we don't want in 2.0
> and we file post 2.0 issues there
> (h) Nutch 2.0 trunk is fixed, and brought up to speed and old code is
> removed. All unit tests should pass regression where it makes sense.
> (i) Nutch documentation is brought up to date on wiki and checked into
> SVN
> (j) We roll a 2.0 release
>
+1
>
> I'd be happy to do (a), (c), (e) and (g) tomorrow, and would like to
> participate in (d) and (f).
I'm thinking Julien and Doğacan would be the
> best people to do (b) and (i).
Doğacan is in the process of writing the documentation
> (h) should be a result of all steps prior
> (a)-(g), and as for (j), I'd be happy to do (j) when the time comes.
>
> So, if I don't hear any objections, I'll do (a), (c), (e) and (g)
> tomorrow... (6/28, likely PM PST Los Angeles time)
cool, thanks
J.
--
DigitalPebble Ltd
Open Source Solutions for Text Engineering
http://www.digitalpebble.com
Re: Nutch 2.0
Posted by MoD <w...@ant.com>.
Hi all,
I agree, we need a clear look on the code base:
What about extracting all patch from git commit and submitting them on a few
tickets.
Using "git-format-patch"
Issues will help having a brief functionality / componant description, and
tracking eventual bugs.
We don't need to know all code base yet.
Regards,
Louis
On Jun 28, 2010 2:30 AM, "Julien Nioche" <li...@gmail.com>
wrote:
Hi guys,
> >>> (a) svn copy NutchBase from GitHub to the nutchbase branch in
> >>> http://svn.apache.org...
I surely agree that this is an issue. My comment was that the code on the
SVN nutchbase branch is quite deprecated and that merging with the GIT
version would not necessarily be the best way of getting a clear picture of
the new architecture. You are right that putting it to trunk is probably a
bit drastic
>
>
> What about an intermediate solution: we delete the svn:nutchbase, import
> the code from gith...
that would be a good option. If we do as you suggested then we would get a
good balance in terms of time vs traceability.
J.
--
DigitalPebble Ltd
Open Source Solutions for Text Engineering
http://www.digitalpebble.com
Re: Nutch 2.0
Posted by Julien Nioche <li...@gmail.com>.
Hi guys,
>>> (a) svn copy NutchBase from GitHub to the nutchbase branch in
> >>> http://svn.apache.org/repos/asf/nutch/branches/nutchbase bringing the
> ASF
> >>> branch up to date.
> >>
> >> this seems like an unnecessary step. There has been an enormous amount
> of
> >> changes between the nutchbase branch and the version on GitHub - pretty
> much
> >> EVERY class has been modified + a lot of classes have been removed
> etc... the
> >> nutchbase branch on svn is completely obsolete so I suggest that simply
> get
> >> rid of the nutchbase branch and move the code to trunk directly (after
> doing c
> >> below of course)
> >
> > That's the problem. If _every_ class has been modified like you state,
> then
> > it's been modified outside of Apache and there is no SVN commit history
> and
> > therefore no public log of the code that's been modified. We need to
> rectify
> > that somehow...
>
> I have to agree with Chris that this is problematic, it's also an issue
> of the Nutch community involvement, and the fact that other committers
> don't really know that code.
>
I surely agree that this is an issue. My comment was that the code on the
SVN nutchbase branch is quite deprecated and that merging with the GIT
version would not necessarily be the best way of getting a clear picture of
the new architecture. You are right that putting it to trunk is probably a
bit drastic
>
> What about an intermediate solution: we delete the svn:nutchbase, import
> the code from github on a branch, and then merge it piecewise according
> to the major areas of functionality that you outlined? I understand that
> this would be more work than importing it wholesale as trunk, but at
> least the process of arriving at the final codebase will be clear.
> Perhaps we could treat this as an investment in the education of the
> rest of the team, which will pay off in the end because other developers
> will be able to catch up sooner.
>
that would be a good option. If we do as you suggested then we would get a
good balance in terms of time vs traceability.
J.
--
DigitalPebble Ltd
Open Source Solutions for Text Engineering
http://www.digitalpebble.com
Re: Nutch 2.0
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Guys,
Just to pass along some info, I've gone the Software Grant route before. AFAIUI, it's there to protect Apache and the contributors, but is mainly used to iron out code where the licensing issues aren't particularly clear. That said, I also understand the Nutchbase at Github is ASL 2.0 licensed (as is GORA). If that's the case, then I think we are OK with simply something like JIRA...
Cheers,
Chris
On 6/28/10 1:10 AM, "Sami Siren" <si...@apache.org> wrote:
On 06/28/2010 10:10 AM, Andrzej Bialecki wrote:
> On 2010-06-28 07:49, Sami Siren wrote:
>> One aspect that has not been discussed yet is the legal aspect.
>> According to http://incubator.apache.org/ip-clearance/index.html
>> there is a formal process for integrating externally development
>> efforts that have happened outside of Apache. Should we be
>> following the ip clearance process in this case too?
>
> The concept of a "substantial contribution" that should be subject to
> a software grant is somewhat tenuous, though. Keep in mind that you
> do something equivalent in JIRA already - when you check the "Grant
> license to ASF" box you perform a micro-grant. So the question is
> whether we should go through a full grant or through the JIRA
> micro-grant.
>
> In my opinion it's ok to do the latter, since much of the code is
> simply a modified version of Nutch classes - not counting GORA, of
> course, but that part will be added as a third-party lib. So IMHO
> it's enough to zip all source (without libs), attach it to a JIRA
> issue and mark the checkbox. Then we follow the process outlined by
> Chris, which imports the same codebase into our svn. What do you
> think?
I do not know what is the right approach, that's why I asked the
question. Also I have not looked at the donation but the following
comment made me think it might fall into "substantial" category:
"
There has been an enormous amount of changes between the nutchbase
branch and the version on GitHub - pretty much EVERY class has been
modified + a lot of classes have been removed etc...
"
> If folks agree that this is sufficient, then Dogacan& Enis - can
> you please create a separate JIRA issue, prepare a patch like this,
> mark the checkbox, and list all dependencies and their licenses for
> those that are not already in Nutch svn?
This would be a good thing to do in any case. It would help to
understand what the donation is about and also help to decide which process
(if any) needs to be followed.
--
Sami Siren
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Nutch 2.0
Posted by Sami Siren <si...@apache.org>.
On 06/28/2010 10:10 AM, Andrzej Bialecki wrote:
> On 2010-06-28 07:49, Sami Siren wrote:
>> One aspect that has not been discussed yet is the legal aspect.
>> According to http://incubator.apache.org/ip-clearance/index.html
>> there is a formal process for integrating externally development
>> efforts that have happened outside of Apache. Should we be
>> following the ip clearance process in this case too?
>
> The concept of a "substantial contribution" that should be subject to
> a software grant is somewhat tenuous, though. Keep in mind that you
> do something equivalent in JIRA already - when you check the "Grant
> license to ASF" box you perform a micro-grant. So the question is
> whether we should go through a full grant or through the JIRA
> micro-grant.
>
> In my opinion it's ok to do the latter, since much of the code is
> simply a modified version of Nutch classes - not counting GORA, of
> course, but that part will be added as a third-party lib. So IMHO
> it's enough to zip all source (without libs), attach it to a JIRA
> issue and mark the checkbox. Then we follow the process outlined by
> Chris, which imports the same codebase into our svn. What do you
> think?
I do not know what is the right approach, that's why I asked the
question. Also I have not looked at the donation but the following
comment made me think it might fall into "substantial" category:
"
There has been an enormous amount of changes between the nutchbase
branch and the version on GitHub - pretty much EVERY class has been
modified + a lot of classes have been removed etc...
"
> If folks agree that this is sufficient, then Dogacan& Enis - can
> you please create a separate JIRA issue, prepare a patch like this,
> mark the checkbox, and list all dependencies and their licenses for
> those that are not already in Nutch svn?
This would be a good thing to do in any case. It would help to
understand what the donation is about and also help to decide which process
(if any) needs to be followed.
--
Sami Siren
Re: Nutch 2.0
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-06-28 07:49, Sami Siren wrote:
> One aspect that has not been discussed yet is the legal aspect.
> According to
> http://incubator.apache.org/ip-clearance/index.html there is a formal
> process for integrating externally development efforts that have
> happened outside of Apache. Should we be following the ip clearance
> process in this case too?
The concept of a "substantial contribution" that should be subject to a
software grant is somewhat tenuous, though. Keep in mind that you do
something equivalent in JIRA already - when you check the "Grant license
to ASF" box you perform a micro-grant. So the question is whether we
should go through a full grant or through the JIRA micro-grant.
In my opinion it's ok to do the latter, since much of the code is simply
a modified version of Nutch classes - not counting GORA, of course, but
that part will be added as a third-party lib. So IMHO it's enough to zip
all source (without libs), attach it to a JIRA issue and mark the
checkbox. Then we follow the process outlined by Chris, which imports
the same codebase into our svn. What do you think?
If folks agree that this is sufficient, then Dogacan & Enis - can you
please create a separate JIRA issue, prepare a patch like this, mark the
checkbox, and list all dependencies and their licenses for those that
are not already in Nutch svn?
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Nutch 2.0
Posted by Sami Siren <si...@apache.org>.
One aspect that has not been discussed yet is the legal aspect.
According to
http://incubator.apache.org/ip-clearance/index.html there is a formal
process for integrating externally development efforts that have
happened outside of Apache. Should we be following the ip clearance
process in this case too?
--
Sami Siren
On 06/28/2010 08:24 AM, Mattmann, Chris A (388J) wrote:
> Hey Guys,
>
>> What about an intermediate solution: we delete the svn:nutchbase, import
>> the code from github on a branch, and then merge it piecewise according
>> to the major areas of functionality that you outlined? I understand that
>> this would be more work than importing it wholesale as trunk, but at
>> least the process of arriving at the final codebase will be clear.
>> Perhaps we could treat this as an investment in the education of the
>> rest of the team, which will pay off in the end because other developers
>> will be able to catch up sooner.
>
> +1.
>
>>
>>> I'm not sure I get it? How does seeing above deal with (c)? In terms of
>>> nutchbase branch merging with trunk, again, I'm a bit worried here since the
>>> three of you (Julien, Enis, and Doğacan) are the only ones that were
>>> significantly involved (correct me if I'm wrong) with the development of
>>> Nutchbase at Github, right, yet there are 7 Nutch PMC members and committers
>>> (one of which does not include Enis). How do you expect us to maintain the
>>> code you bring over unless those of us that were not involved in the Github
>>> development have some history/notion of what's been done vis-à-vis Github
>>> and Apache SVN?
>>
>> There are precedents for this, e.g. a wholesale import of mapred+NDFS by
>> Doug and Mike C... but I have to say that it took a _very_ long time (1
>> year?) for other developers to get up to speed with the new code base...
>> and it could've been a failure if not for a steady involvement of Doug
>> throughout this process and long thereafter.
>
> Wise to point out, Andrzej.
>
> Okey dok, I'm +1 for:
>
> (a) deleting svn:nutchbase
> (b) svn:importing Git Nutchbase.
> (c) branch current 1.2-trunk as 1.2-branch
> (d) iteratively apply patches from new svn:nutchbase to trunk to bring
> it up to snuff.
> (e) roll the version # in nutch trunk to 2.0-dev
> (f) all issues in JIRA should be updated to reflect 2.0-dev fixes where
> it makes sense
> (g) a 2.1 version is added to mark anything that we don't want in 2.0
> and we file post 2.0 issues there
> (h) Nutch 2.0 trunk is fixed, and brought up to speed and old code is
> removed. All unit tests should pass regression where it makes sense.
> (i) Nutch documentation is brought up to date on wiki and checked into
> SVN
> (j) We roll a 2.0 release
>
> I'd be happy to do (a), (c), (e) and (g) tomorrow, and would like to
> participate in (d) and (f). I'm thinking Julien and Doğacan would be the
> best people to do (b) and (i). (h) should be a result of all steps prior
> (a)-(g), and as for (j), I'd be happy to do (j) when the time comes.
>
> So, if I don't hear any objections, I'll do (a), (c), (e) and (g)
> tomorrow... (6/28, likely PM PST Los Angeles time)
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: Chris.Mattmann@jpl.nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
Re: Nutch 2.0
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Guys,
> What about an intermediate solution: we delete the svn:nutchbase, import
> the code from github on a branch, and then merge it piecewise according
> to the major areas of functionality that you outlined? I understand that
> this would be more work than importing it wholesale as trunk, but at
> least the process of arriving at the final codebase will be clear.
> Perhaps we could treat this as an investment in the education of the
> rest of the team, which will pay off in the end because other developers
> will be able to catch up sooner.
+1.
>
>> I'm not sure I get it? How does seeing above deal with (c)? In terms of
>> nutchbase branch merging with trunk, again, I'm a bit worried here since the
>> three of you (Julien, Enis, and Doğacan) are the only ones that were
>> significantly involved (correct me if I'm wrong) with the development of
>> Nutchbase at Github, right, yet there are 7 Nutch PMC members and committers
>> (one of which does not include Enis). How do you expect us to maintain the
>> code you bring over unless those of us that were not involved in the Github
>> development have some history/notion of what's been done vis-à-vis Github
>> and Apache SVN?
>
> There are precedents for this, e.g. a wholesale import of mapred+NDFS by
> Doug and Mike C... but I have to say that it took a _very_ long time (1
> year?) for other developers to get up to speed with the new code base...
> and it could've been a failure if not for a steady involvement of Doug
> throughout this process and long thereafter.
Wise to point out, Andrzej.
Okey dok, I'm +1 for:
(a) deleting svn:nutchbase
(b) svn:importing Git Nutchbase.
(c) branch current 1.2-trunk as 1.2-branch
(d) iteratively apply patches from new svn:nutchbase to trunk to bring
it up to snuff.
(e) roll the version # in nutch trunk to 2.0-dev
(f) all issues in JIRA should be updated to reflect 2.0-dev fixes where
it makes sense
(g) a 2.1 version is added to mark anything that we don't want in 2.0
and we file post 2.0 issues there
(h) Nutch 2.0 trunk is fixed, and brought up to speed and old code is
removed. All unit tests should pass regression where it makes sense.
(i) Nutch documentation is brought up to date on wiki and checked into
SVN
(j) We roll a 2.0 release
I'd be happy to do (a), (c), (e) and (g) tomorrow, and would like to
participate in (d) and (f). I'm thinking Julien and Doğacan would be the
best people to do (b) and (i). (h) should be a result of all steps prior
(a)-(g), and as for (j), I'd be happy to do (j) when the time comes.
So, if I don't hear any objections, I'll do (a), (c), (e) and (g)
tomorrow... (6/28, likely PM PST Los Angeles time)
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Nutch 2.0
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-06-27 17:34, Mattmann, Chris A (388J) wrote:
> Hi Julien,
>
>>> (a) svn copy NutchBase from GitHub to the nutchbase branch in
>>> http://svn.apache.org/repos/asf/nutch/branches/nutchbase bringing the ASF
>>> branch up to date.
>>
>> this seems like an unnecessary step. There has been an enormous amount of
>> changes between the nutchbase branch and the version on GitHub - pretty much
>> EVERY class has been modified + a lot of classes have been removed etc... the
>> nutchbase branch on svn is completely obsolete so I suggest that simply get
>> rid of the nutchbase branch and move the code to trunk directly (after doing c
>> below of course)
>
> That's the problem. If _every_ class has been modified like you state, then
> it's been modified outside of Apache and there is no SVN commit history and
> therefore no public log of the code that's been modified. We need to rectify
> that somehow...
I have to agree with Chris that this is problematic, it's also an issue
of the Nutch community involvement, and the fact that other committers
don't really know that code.
What about an intermediate solution: we delete the svn:nutchbase, import
the code from github on a branch, and then merge it piecewise according
to the major areas of functionality that you outlined? I understand that
this would be more work than importing it wholesale as trunk, but at
least the process of arriving at the final codebase will be clear.
Perhaps we could treat this as an investment in the education of the
rest of the team, which will pay off in the end because other developers
will be able to catch up sooner.
> I'm not sure I get it? How does seeing above deal with (c)? In terms of
> nutchbase branch merging with trunk, again, I'm a bit worried here since the
> three of you (Julien, Enis, and Doğacan) are the only ones that were
> significantly involved (correct me if I'm wrong) with the development of
> Nutchbase at Github, right, yet there are 7 Nutch PMC members and committers
> (one of which does not include Enis). How do you expect us to maintain the
> code you bring over unless those of us that were not involved in the Github
> development have some history/notion of what's been done vis-à-vis Github
> and Apache SVN?
There are precedents for this, e.g. a wholesale import of mapred+NDFS by
Doug and Mike C... but I have to say that it took a _very_ long time (1
year?) for other developers to get up to speed with the new code base...
and it could've been a failure if not for a steady involvement of Doug
throughout this process and long thereafter.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Nutch 2.0
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Julien,
>> (a) svn copy NutchBase from GitHub to the nutchbase branch in
>> http://svn.apache.org/repos/asf/nutch/branches/nutchbase bringing the ASF
>> branch up to date.
>
> this seems like an unnecessary step. There has been an enormous amount of
> changes between the nutchbase branch and the version on GitHub - pretty much
> EVERY class has been modified + a lot of classes have been removed etc... the
> nutchbase branch on svn is completely obsolete so I suggest that simply get
> rid of the nutchbase branch and move the code to trunk directly (after doing c
> below of course)
That's the problem. If _every_ class has been modified like you state, then
it's been modified outside of Apache and there is no SVN commit history and
therefore no public log of the code that's been modified. We need to rectify
that somehow...
>> (b) Once the GORA license issues are figured out (they must be
>> compatible with the ASF or we cannot use it), then we update Nutch to depend
>> on the GORA jars via Ivy?
>
> see comment above about the license.
Right, the license is one part of it, the jars are the other. Does GORA
produce a jar file?
>> (c) svn tag current Nutch trunk as 1.2-branch
>> (d) svn merge nutchbase branch with nutch trunk
>
> see above
I'm not sure I get it? How does seeing above deal with (c)? In terms of
nutchbase branch merging with trunk, again, I'm a bit worried here since the
three of you (Julien, Enis, and Doğacan) are the only ones that were
significantly involved (correct me if I'm wrong) with the development of
Nutchbase at Github, right, yet there are 7 Nutch PMC members and committers
(one of which does not include Enis). How do you expect us to maintain the
code you bring over unless those of us that were not involved in the Github
development have some history/notion of what's been done vis-à-vis Github
and Apache SVN?
>
>
>> (e) roll the version # in nutch trunk to 2.0-dev
>> (f) all issues in JIRA should be updated to reflect 2.0-dev fixes where
>> it makes sense
>> (g) a 2.1 version is added to mark anything that we don't want in 2.0
>> and we file post 2.0 issues there
>> (h) Nutch 2.0 trunk is fixed, and brought up to speed and old code is
>> removed. All unit tests should pass regression where it makes sense.
>> (i) Nutch documentation is brought up to date on wiki and checked into
>> SVN
>> (j) We roll a 2.0 release
>
> OK with everything else
Great. I'll wait to hear back on my questions/comments above.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Nutch 2.0
Posted by Julien Nioche <li...@gmail.com>.
Hi,
a) is GORA ASL licensed?
>
it is. see http://github.com/enis/gora/blob/master/LICENSE.txt
> b) what's the maintenance plan for GORA? Will it continue to live in
> Github? Will you guys propose it into the Apache Incubator as an ASF
> project?
>
I confirm that the plan is definitely to move it to the incubator or
possibly as a Hadoop subproject.
>
> (a) svn copy NutchBase from GitHub to the nutchbase branch in
> http://svn.apache.org/repos/asf/nutch/branches/nutchbase bringing the ASF
> branch up to date.
>
this seems like an unnecessary step. There has been an enormous amount of
changes between the nutchbase branch and the version on GitHub - pretty much
EVERY class has been modified + a lot of classes have been removed etc...
the nutchbase branch on svn is completely obsolete so I suggest that simply
get rid of the nutchbase branch and move the code to trunk directly (after
doing c below of course)
> (b) Once the GORA license issues are figured out (they must be
> compatible with the ASF or we cannot use it), then we update Nutch to
> depend
> on the GORA jars via Ivy?
>
see comment above about the license.
> (c) svn tag current Nutch trunk as 1.2-branch
> (d) svn merge nutchbase branch with nutch trunk
>
see above
> (e) roll the version # in nutch trunk to 2.0-dev
> (f) all issues in JIRA should be updated to reflect 2.0-dev fixes where
> it makes sense
> (g) a 2.1 version is added to mark anything that we don't want in 2.0
> and we file post 2.0 issues there
> (h) Nutch 2.0 trunk is fixed, and brought up to speed and old code is
> removed. All unit tests should pass regression where it makes sense.
> (i) Nutch documentation is brought up to date on wiki and checked into
> SVN
> (j) We roll a 2.0 release
>
OK with everything else
Julien
--
DigitalPebble Ltd
Open Source Solutions for Text Engineering
http://www.digitalpebble.com