You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Doug Cutting <cu...@nutch.org> on 2005/03/23 00:49:35 UTC
Re: Licenses
Andy Hedges wrote:
> Was just checking through the mailing list to see how things were
> going with Nutch. It seems that two of my contributions had to be
> removed because they aren't compatible with the ASF licenses (rtf and
> mp3 parsers). So what needs to be done to get them back in? Do
> alternative libraries need to be written/used that are either BSD or ASF
> licensed? Could you let me know and I'll try and find sometime to
> rectify the issue.
Right, Apache doesn't permit GPL'd stuff on Apache servers. One
possibility is to rewrite them to use different libraries. Another
approach is to just get the user to download the required stuff themselves.
For example, a plugin's build.xml can download required libraries as
follows. First, add a few properties noting where the required jar
lives on the net, and where it should be stored:
<property name="foo.jar" value="lib/my.jar"/>
<property name="foo.url" value="http://foo.com/foo.jar"/>
http://ant.apache.org/manual/CoreTasks/available.html
Then define a target to download it as needed:
<available file="${foo.jar}" property="foo.jar.available"/>
<target name="download-foo" unless="foo.jar.available">
<get src="${foo.url}" dest="${foo.jar}"/>
</target>
http://ant.apache.org/manual/CoreTasks/available.html
http://ant.apache.org/manual/CoreTasks/get.html
and finally, override the init-plugin task to first download it:
<target name="init-plugin" depends="download-libs"/>
Does this make sense?
Doug
Re: Licenses
Posted by Doug Cutting <cu...@nutch.org>.
Hari Kodungallur wrote:
> I also noticed that the file:// and ftp:// protocol plugins are
> disabled because it uses the activation.jar (I think Sun's jar files
> are not redistributable).
> I looked into the files and all it uses it the MimetypeFileType and
> MimeType classes from the activation.jar. I have seen that some other
> apache projects (jetspeed, turbine etc) use their own, sometime
> simplistic, implementation of MimeType mapping. I am wondering whether
> nutch also plans to do that? I can provide you with patches if that is
> the case.
That would be great.
Thanks,
Doug
Re: Licenses
Posted by Doug Cutting <cu...@nutch.org>.
Hari Kodungallur wrote:
> Does nutch have a requirement to be compatibel with JDK 1.3.x?
No. Nutch is meant to work with JDK 1.4+.
Doug
Re: Licenses
Posted by Hari Kodungallur <ha...@gmail.com>.
Does nutch have a requirement to be compatibel with JDK 1.3.x?
I would like to use the java.nio.* package, but I wanted to find out
about the requirement before it is too late :)
thanks
-Hari
On Wed, 23 Mar 2005 23:47:47 -0800, John X <jo...@neasys.com> wrote:
> On Wed, Mar 23, 2005 at 10:36:12PM -0800, Hari Kodungallur wrote:
> > Hi John,
> >
> > I will do them. No problems.
> > But one question: for (3) is there any other way other than reading
> > the /usr/share/magic.mime file? I am curious whether there is a
> > platform independent way which I am not aware of...
>
> Maybe have it searched/loaded from standard places,
> including nutch's conf dir?
>
> John
>
Re: Licenses
Posted by John X <jo...@neasys.com>.
On Wed, Mar 23, 2005 at 10:36:12PM -0800, Hari Kodungallur wrote:
> Hi John,
>
> I will do them. No problems.
> But one question: for (3) is there any other way other than reading
> the /usr/share/magic.mime file? I am curious whether there is a
> platform independent way which I am not aware of...
Maybe have it searched/loaded from standard places,
including nutch's conf dir?
John
>
> thanks
> -Hari
>
>
> > Ideally we need something that provides mapping among:
> > (1) filename extension
> > (2) mime type
> > (3) file magic number (unix 'file' command)
> > jaf only gives (1) -> (2) one way.
> > It will be really great if yours can do (3) too.
> > Thanks,
> >
> > John
> >
>
__________________________________________
http://www.neasys.com - A Good Place to Be
Come to visit us today!
Re: Licenses
Posted by Hari Kodungallur <ha...@gmail.com>.
Hi John,
I will do them. No problems.
But one question: for (3) is there any other way other than reading
the /usr/share/magic.mime file? I am curious whether there is a
platform independent way which I am not aware of...
thanks
-Hari
> Ideally we need something that provides mapping among:
> (1) filename extension
> (2) mime type
> (3) file magic number (unix 'file' command)
> jaf only gives (1) -> (2) one way.
> It will be really great if yours can do (3) too.
> Thanks,
>
> John
>
Wiki has been moved....
Posted by Chirag Chaman <de...@filangy.com>.
Yes, it's finally moved. But, PLEASE DON'T POST ANYTHING NEW YET.
Please treat this as DRAFT 1. I'm sure Doug or one of the more seasoned
folks will find something wrong and it might be easier for me to make
changes to the script and repost everything. Thus, let's wait for an "All
Clear" from one of them.
NOTE:
1. A lot of the links are broken, namely all those that were not in
WikiStandardFormat. I tried my best to keep all the other links that had
full path names, but a few dropped off during the port. Thus, please check
the pages and feel free to make change/edits as you see fit.
2. User pages were not ported over
3. Main page (FrontPage) has not been changed, so the best way to see all
the pages is via the RecentChanges
4. Right now all of the pages are at the Top level, if this needs to be
changed please let me know. This should be an easy change to the script.
5. The big page to fix with links is "Nutch" -- which should become the new
FrontPage. So, that should be the first page to fix.
Please feel free to email me with any questions or if you'd like the sources
to the scripts (it's ugly but gets the job done).
Thankx
CC-
--------------------------------------------
Filangy, Inc.
Interested in Improving Search? Join our Team!
http://filangy.com/jointheteam.jsp
RE: [Nutch-dev] Converted Wiki
Posted by Chirag Chaman <de...@filangy.com>.
Luke,
Thankx -- this was helpful.
I was missing a cookie jar handler in my script
CC-
-----Original Message-----
From: nutch-developers-admin@lists.sourceforge.net
[mailto:nutch-developers-admin@lists.sourceforge.net] On Behalf Of Luke
Baker
Sent: Friday, April 08, 2005 4:48 PM
To: nutch-dev@incubator.apache.org
Subject: Re: [Nutch-dev] Converted Wiki
On 04/08/2005 04:27 PM, Chirag Chaman wrote:
> So, I'm all ready to post the new wiki....but hit a little snafu.
>
> Basically, I can't seem to login via my script. Have a Perl script
> that needs to do the following:
>
> 1. Login
Chirag,
Was your script able to login previously? It needs to keep track of cookies
(not discard them), so that it'll send them on each subsequent request.
Once that is setup, I'd imagine you'd just have to hit the login page [1]
with the following POST information:
action=userform&username=UserName&password=yourpasswordhere&password2=&email
=&login=Login
Subsequent requests would have to use the same object so that they have
access to the cookie(s), I think. Depends on how you implemented it.
I don't know if this is plainly obvious or what you're looking for in
regards for help. A snippet of the login code might be helpful.
Luke
[1] http://wiki.apache.org/nutch/UserPreferences
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide Read honest & candid reviews
on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Nutch-developers mailing list
Nutch-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nutch-developers
Re: [Nutch-dev] Converted Wiki
Posted by Luke Baker <lu...@gospelcom.net>.
On 04/08/2005 04:27 PM, Chirag Chaman wrote:
> So, I'm all ready to post the new wiki....but hit a little snafu.
>
> Basically, I can't seem to login via my script. Have a Perl script that
> needs to do the following:
>
> 1. Login
Chirag,
Was your script able to login previously? It needs to keep track of
cookies (not discard them), so that it'll send them on each subsequent
request. Once that is setup, I'd imagine you'd just have to hit the
login page [1] with the following POST information:
action=userform&username=UserName&password=yourpasswordhere&password2=&email=&login=Login
Subsequent requests would have to use the same object so that they have
access to the cookie(s), I think. Depends on how you implemented it.
I don't know if this is plainly obvious or what you're looking for in
regards for help. A snippet of the login code might be helpful.
Luke
[1] http://wiki.apache.org/nutch/UserPreferences
RE: [Nutch-dev] Converted Wiki
Posted by Chirag Chaman <de...@filangy.com>.
So, I'm all ready to post the new wiki....but hit a little snafu.
Basically, I can't seem to login via my script. Have a Perl script that
needs to do the following:
1. Login
Then, for each Topic
2a. Edit the topic
2b. save the new file
Part 2 is working fine (checked via hand by pasting the URL in browser), but
the logging in part does not seem to be working.
Anyone have any ideas?
CC-
-----Original Message-----
From: Doug Cutting [mailto:cutting@nutch.org]
Sent: Thursday, April 07, 2005 1:03 PM
To: nutch-dev@incubator.apache.org
Subject: Re: [Nutch-dev] Converted Wiki
Chirag Chaman wrote:
> if by Friday there are no "raised hands" or objections....the old Wiki
> will be moved over to Moin Moin, over the weekend.
>
> Twiki markups will be converted to Moin Moin...and HTML to readable text.
+1
Thanks! With this the move to Apache will be complete!
Doug
Re: [Nutch-dev] Converted Wiki
Posted by Doug Cutting <cu...@nutch.org>.
Chirag Chaman wrote:
> if by Friday there are no "raised hands" or
> objections....the old Wiki will be moved over to Moin Moin, over the
> weekend.
>
> Twiki markups will be converted to Moin Moin...and HTML to readable text.
+1
Thanks! With this the move to Apache will be complete!
Doug
RE: [Nutch-dev] Converted Wiki
Posted by Chirag Chaman <de...@filangy.com>.
I'm out of pocket this week, but if by Friday there are no "raised hands" or
objections....the old Wiki will be moved over to Moin Moin, over the
weekend.
Twiki markups will be converted to Moin Moin...and HTML to readable text.
CC-
-----Original Message-----
From: nutch-developers-admin@lists.sourceforge.net
[mailto:nutch-developers-admin@lists.sourceforge.net] On Behalf Of Chirag
Chaman
Sent: Sunday, April 03, 2005 2:22 PM
To: nutch-dev@incubator.apache.org
Subject: [Nutch-dev] Converted Wiki
Folks,
I was just about to post the converted Wiki over to apache when I decide to
so a small test...And it seems I would have overwritten the new WebDB file.
I see that quite a few files have been added to the new wiki. I don't want
to overwrite files -- so if you are working on porting a file over can you
let me know the names and I'll exclude them from posting.
Again, let's coordinate our efforts. I'm standing down until I get a
confirmation of whose working on what w.r.t moving over pages to Nutch.
Please let me know the pages you would like to port and they will be
excluded.
Alternately, I can post the converted pages on an FTP site, and the original
authors can grab them from there, make changes as needed and post them to
the new wiki.
Regards,
CC
--------------------------------------------
Filangy, Inc.
Interested in Improving Search? Join our Team!
http://filangy.com/jointheteam.jsp
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide Read honest & candid reviews
on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Nutch-developers mailing list
Nutch-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nutch-developers
Converted Wiki
Posted by Chirag Chaman <de...@filangy.com>.
Folks,
I was just about to post the converted Wiki over to apache when I decide to
so a small test...And it seems I would have overwritten the new WebDB file.
I see that quite a few files have been added to the new wiki. I don't want
to overwrite files -- so if you are working on porting a file over can you
let me know the names and I'll exclude them from posting.
Again, let's coordinate our efforts. I'm standing down until I get a
confirmation of whose working on what w.r.t moving over pages to Nutch.
Please let me know the pages you would like to port and they will be
excluded.
Alternately, I can post the converted pages on an FTP site, and the original
authors can grab them from there, make changes as needed and post them to
the new wiki.
Regards,
CC
--------------------------------------------
Filangy, Inc.
Interested in Improving Search? Join our Team!
http://filangy.com/jointheteam.jsp
Re: Licenses
Posted by Hari Kodungallur <ha...@gmail.com>.
On Apr 3, 2005 1:16 PM, Jérôme Charron <je...@gmail.com> wrote:
>
> The best process to avoid such situation is to create issues on jira. Isn't
> it?
> Jira must be the referencial for new features, bug, fixes, patches. No?
>
No question.
Re: Licenses
Posted by Jérôme Charron <je...@gmail.com>.
> Seems like Jerome has spent quite a bit of his time and effort into
> this. We can just take his contribution for the mime/magic mapper. I
> have no issues with that. The last thing we need is to review two
> pieces of code doing the same thing.
The best process to avoid such situation is to create issues on jira. Isn't
it?
Jira must be the referencial for new features, bug, fixes, patches. No?
Regards
Jerome
--
http://motrech.free.fr/
http://frutch.free.fr/
Fwd: Licenses
Posted by Hari Kodungallur <ha...@gmail.com>.
forwarding to the group...
---------- Forwarded message ----------
From: Hari Kodungallur <ha...@gmail.com>
Date: Apr 3, 2005 11:58 AM
Subject: Re: Licenses
To: Jérôme Charron <je...@gmail.com>
Jerome,
I think it was mentioned in this thread that I was going to work on it.
All,
Seems like Jerome has spent quite a bit of his time and effort into
this. We can just take his contribution for the mime/magic mapper. I
have no issues with that. The last thing we need is to review two
pieces of code doing the same thing.
rgds
-Hari
On Apr 3, 2005 5:30 AM, Jérôme Charron <je...@gmail.com> wrote:
> Hello,
>
> I'm very disapointed by this mail.
> In fact, I annouce, two weeks ago that I was currently working on a
> Extension/MimeType mapper and MagicNumber/MimeTye mapper.
> And I created an issue related to this point too, in which I mentionned I
> was working on it !!
> My implementation is now ready to be delivered in a few days for the Nutch
> community (just need to perform some additional Unit Tests).
> I really hate loosing my time and working for nothing, especially if it is
> due to a lack of communication.
> So please, in the future, announce that you are working on a topic!!! (it
> could be certainly better to combine our efforts instead of doing the same
> work twice!).
> So, I will send my contribution in few days.....
>
> Jerome
Re: Licenses
Posted by Jérôme Charron <je...@gmail.com>.
Hello,
I'm very disapointed by this mail.
In fact, I annouce, two weeks ago that I was currently working on a
Extension/MimeType mapper and MagicNumber/MimeTye mapper.
And I created an issue related to this point too, in which I mentionned I
was working on it !!
My implementation is now ready to be delivered in a few days for the Nutch
community (just need to perform some additional Unit Tests).
I really hate loosing my time and working for nothing, especially if it is
due to a lack of communication.
So please, in the future, announce that you are working on a topic!!! (it
could be certainly better to combine our efforts instead of doing the same
work twice!).
So, I will send my contribution in few days.....
Jerome
On Apr 3, 2005 10:59 AM, Hari Kodungallur <ha...@gmail.com>
wrote:
>
> Oops.. forgot to do "reply all". Seding again to the group.
>
> John, please ignore the attachment in my prvs email, use this one
> instead (an extra backup file got into the other one)
>
> ------------------------------------------------
>
> Here's a first version of the mime-mapper and magic-mime-mapper. i
> have packaged it at org.apache.nutch.util.mime. None of the files
> that depend on mime mapping is changed. This is just a set of utility
> classes for magic/mime mapping. We can make those changes after this
> code is verified and checked in.
>
> There are some TODOs that I have documented in the MagicMimeEntry
> file. Also I need to write a proper test case. If I need to parse some
> sample files (pdf, doc, mp3 etc) for use in the test cases, what is
> the procedure to do it? Just copy some samples and check them in and
> then use it in the test case? (I used hard-coded paths to my local
> disk for the tests; since that won't work anywhere else, I commented
> out that test case).
> Also, now the mime mapper (mapping based on file extension) and magic
> mime mapper (mapping from file contents based on magic.mime file) are
> two different utility classes. They can be combined, if needed, to
> provided a single interface.
>
> Please take a look at the files and let me know if there are any
> issues. After your review, if its okay, can you check them in? I will
> work on modifying some plugins that use the mime mapper and also work
> on the TODOs, test cases, code documentation etc
>
> thx
> -Hari
>
> > On Mar 23, 2005 10:22 AM, John X <jo...@neasys.com> wrote:
> > > On Wed, Mar 23, 2005 at 12:35:30AM -0800, Hari Kodungallur wrote:
> > > > On Wed, 23 Mar 2005 00:51:53 -0800, John X <jo...@neasys.com> wrote:
> > > > >
> > > > > It will be great if you can help on that. Plugin index-more also
> uses it.
> > > > > I know there are two opensource efforts:
> > > > > http://jafi.sourceforge.net/maven-reports.html
> > > > > http://www.gnu.org/software/classpathx/jaf/jaf.html
> > > > > The gnu one is out of the question now.
> > > > > I am currently short of time, so any help will be greatly
> appreciated.
> > > > >
> > > > > One interesting observation: there is an activation.jar
> > > > > (under ./common/lib/) in jakarta-tomcat-4.1.31.tar.gz
> > > > > We need to find out which one this is?
> > > > >
> > > >
> > > > Hm.. it is interesting to note that the activation.jar file in
> tomcat
> > > > is Sun's. Is that because some of the code to tomcat was donated by
> > > > Sun?
> > > >
> > > > In any case, I will start some work on the MimeTypes implementation
> as
> > > > needed by the file, ftp and index-more plugins and provide you with
> > > > patches. We do not need as exhaustive implementation as Sun's, I
> > > > presume.
> > >
> > > Ideally we need something that provides mapping among:
> > > (1) filename extension
> > > (2) mime type
> > > (3) file magic number (unix 'file' command)
> > > jaf only gives (1) -> (2) one way.
> > > It will be really great if yours can do (3) too.
> > > Thanks,
> > >
> > > John
> > >
> >
> >
> >
>
>
>
--
http://motrech.free.fr/
http://frutch.free.fr/
Re: Licenses
Posted by Hari Kodungallur <ha...@gmail.com>.
Oops.. forgot to do "reply all". Seding again to the group.
John, please ignore the attachment in my prvs email, use this one
instead (an extra backup file got into the other one)
------------------------------------------------
Here's a first version of the mime-mapper and magic-mime-mapper. i
have packaged it at org.apache.nutch.util.mime. None of the files
that depend on mime mapping is changed. This is just a set of utility
classes for magic/mime mapping. We can make those changes after this
code is verified and checked in.
There are some TODOs that I have documented in the MagicMimeEntry
file. Also I need to write a proper test case. If I need to parse some
sample files (pdf, doc, mp3 etc) for use in the test cases, what is
the procedure to do it? Just copy some samples and check them in and
then use it in the test case? (I used hard-coded paths to my local
disk for the tests; since that won't work anywhere else, I commented
out that test case).
Also, now the mime mapper (mapping based on file extension) and magic
mime mapper (mapping from file contents based on magic.mime file) are
two different utility classes. They can be combined, if needed, to
provided a single interface.
Please take a look at the files and let me know if there are any
issues. After your review, if its okay, can you check them in? I will
work on modifying some plugins that use the mime mapper and also work
on the TODOs, test cases, code documentation etc
thx
-Hari
> On Mar 23, 2005 10:22 AM, John X <jo...@neasys.com> wrote:
> > On Wed, Mar 23, 2005 at 12:35:30AM -0800, Hari Kodungallur wrote:
> > > On Wed, 23 Mar 2005 00:51:53 -0800, John X <jo...@neasys.com> wrote:
> > > >
> > > > It will be great if you can help on that. Plugin index-more also uses it.
> > > > I know there are two opensource efforts:
> > > > http://jafi.sourceforge.net/maven-reports.html
> > > > http://www.gnu.org/software/classpathx/jaf/jaf.html
> > > > The gnu one is out of the question now.
> > > > I am currently short of time, so any help will be greatly appreciated.
> > > >
> > > > One interesting observation: there is an activation.jar
> > > > (under ./common/lib/) in jakarta-tomcat-4.1.31.tar.gz
> > > > We need to find out which one this is?
> > > >
> > >
> > > Hm.. it is interesting to note that the activation.jar file in tomcat
> > > is Sun's. Is that because some of the code to tomcat was donated by
> > > Sun?
> > >
> > > In any case, I will start some work on the MimeTypes implementation as
> > > needed by the file, ftp and index-more plugins and provide you with
> > > patches. We do not need as exhaustive implementation as Sun's, I
> > > presume.
> >
> > Ideally we need something that provides mapping among:
> > (1) filename extension
> > (2) mime type
> > (3) file magic number (unix 'file' command)
> > jaf only gives (1) -> (2) one way.
> > It will be really great if yours can do (3) too.
> > Thanks,
> >
> > John
> >
>
>
>
Re: Licenses
Posted by John X <jo...@neasys.com>.
On Wed, Mar 23, 2005 at 12:35:30AM -0800, Hari Kodungallur wrote:
> On Wed, 23 Mar 2005 00:51:53 -0800, John X <jo...@neasys.com> wrote:
> >
> > It will be great if you can help on that. Plugin index-more also uses it.
> > I know there are two opensource efforts:
> > http://jafi.sourceforge.net/maven-reports.html
> > http://www.gnu.org/software/classpathx/jaf/jaf.html
> > The gnu one is out of the question now.
> > I am currently short of time, so any help will be greatly appreciated.
> >
> > One interesting observation: there is an activation.jar
> > (under ./common/lib/) in jakarta-tomcat-4.1.31.tar.gz
> > We need to find out which one this is?
> >
>
> Hm.. it is interesting to note that the activation.jar file in tomcat
> is Sun's. Is that because some of the code to tomcat was donated by
> Sun?
>
> In any case, I will start some work on the MimeTypes implementation as
> needed by the file, ftp and index-more plugins and provide you with
> patches. We do not need as exhaustive implementation as Sun's, I
> presume.
Ideally we need something that provides mapping among:
(1) filename extension
(2) mime type
(3) file magic number (unix 'file' command)
jaf only gives (1) -> (2) one way.
It will be really great if yours can do (3) too.
Thanks,
John
>
> Thanks
> -Hari
>
__________________________________________
http://www.neasys.com - A Good Place to Be
Come to visit us today!
Re: Licenses
Posted by Hari Kodungallur <ha...@gmail.com>.
On Wed, 23 Mar 2005 00:51:53 -0800, John X <jo...@neasys.com> wrote:
>
> It will be great if you can help on that. Plugin index-more also uses it.
> I know there are two opensource efforts:
> http://jafi.sourceforge.net/maven-reports.html
> http://www.gnu.org/software/classpathx/jaf/jaf.html
> The gnu one is out of the question now.
> I am currently short of time, so any help will be greatly appreciated.
>
> One interesting observation: there is an activation.jar
> (under ./common/lib/) in jakarta-tomcat-4.1.31.tar.gz
> We need to find out which one this is?
>
Hm.. it is interesting to note that the activation.jar file in tomcat
is Sun's. Is that because some of the code to tomcat was donated by
Sun?
In any case, I will start some work on the MimeTypes implementation as
needed by the file, ftp and index-more plugins and provide you with
patches. We do not need as exhaustive implementation as Sun's, I
presume.
Thanks
-Hari
Re: Licenses
Posted by John X <jo...@neasys.com>.
On Tue, Mar 22, 2005 at 04:02:17PM -0800, Hari Kodungallur wrote:
> Doug,
>
> I also noticed that the file:// and ftp:// protocol plugins are
> disabled because it uses the activation.jar (I think Sun's jar files
> are not redistributable).
> I looked into the files and all it uses it the MimetypeFileType and
> MimeType classes from the activation.jar. I have seen that some other
> apache projects (jetspeed, turbine etc) use their own, sometime
> simplistic, implementation of MimeType mapping. I am wondering whether
> nutch also plans to do that? I can provide you with patches if that is
> the case.
It will be great if you can help on that. Plugin index-more also uses it.
I know there are two opensource efforts:
http://jafi.sourceforge.net/maven-reports.html
http://www.gnu.org/software/classpathx/jaf/jaf.html
The gnu one is out of the question now.
I am currently short of time, so any help will be greatly appreciated.
One interesting observation: there is an activation.jar
(under ./common/lib/) in jakarta-tomcat-4.1.31.tar.gz
We need to find out which one this is?
John
>
> Aside: it would have been nice if there was a Mimetype mapping
> implementation in some apache commons project.
>
> Thanks
> -Hari
>
__________________________________________
http://www.neasys.com - A Good Place to Be
Come to visit us today!
Re: Licenses
Posted by Hari Kodungallur <ha...@gmail.com>.
Doug,
I also noticed that the file:// and ftp:// protocol plugins are
disabled because it uses the activation.jar (I think Sun's jar files
are not redistributable).
I looked into the files and all it uses it the MimetypeFileType and
MimeType classes from the activation.jar. I have seen that some other
apache projects (jetspeed, turbine etc) use their own, sometime
simplistic, implementation of MimeType mapping. I am wondering whether
nutch also plans to do that? I can provide you with patches if that is
the case.
Aside: it would have been nice if there was a Mimetype mapping
implementation in some apache commons project.
Thanks
-Hari
Re: [Nutch-dev] Re: Licenses
Posted by Doug Cutting <cu...@nutch.org>.
Andy Hedges wrote:
> So I've modified the ant file and it works in the following way. If the
> file hasn't been downloaded it does so. Even if the src has been
> download it may not have been compiled and jared and so that is checked
> and if necessary it is build. It then removes all the intermediate files
> that were created (excluding the src).
>
> Anyway here is the new build.xml
Looks good. I just committed it.
On a related note, I see that JavaCC is now available under a BSD
license! So we could bundle JavaCC with Nutch. Since it's not needed
at runtime, perhaps we should put it in a lib/tools directory. Then we
can freely call it both here and in the top-level build.xml, and need no
longer commit generated code, but rather generate it all at build time...
Doug
Re: [Nutch-dev] Re: Licenses
Posted by Andy Hedges <nu...@hedges.net>.
Doug Cutting wrote:
> Andy Hedges wrote:
>
>> As you can see for rtf I need to do more than just download the jar
>> file as there is not a precompiled version on the net. I have to
>> download the source and then run a number of steps to get to the
>> resulting jar. I'm not sure that the plugin's build.xml is the place to
>> be doing this as it seems to be called twice.
>
>
> The ant excerpts I sent were careful to only download things once.
>
> http://www.mail-archive.com/nutch-developers%40lists.sourceforge.net/msg03602.html
>
>
> Conditionals are not straightforward in Ant. First, <available/> is
> used to note whether things have already been downloaded. Then an
> "unless" attribute is used on the target that actually downloads
> things so that the target is conditioned on the results of the
> <available/>. Finally, override the "init-plugin" target to depend on
> the download task. You could unpack & build the RTF parser in either
> the download or the init-plugin task, as you wish.
>
> Does that make sense?
Yes, it does. I was just concerned that the whole thing was getting
called twice in a single run. I'm used to ant build files where
everything is in targets (excluding classpath setup) and so it is
impossible for anything to be called twice in the same run. I missed the
fact that you could specify a download or init-plugin method and that
would get called. The available method you sent I understood. Your
method ensures that that the files are downloaded once over multiple
runs of the build - I hadn't got that far yet ;). On reflection I can
see that one could used the available/unless method to ensure it doesn't
run twice in the same run - I'm just used to using targets and depends
for that.
So I've modified the ant file and it works in the following way. If the
file hasn't been downloaded it does so. Even if the src has been
download it may not have been compiled and jared and so that is checked
and if necessary it is build. It then removes all the intermediate files
that were created (excluding the src).
Anyway here is the new build.xml
---
<?xml version="1.0"?>
<project name="parse-rtf" default="jar">
<import file="../build-plugin.xml"/>
<property name="rtf-src.jar" value="tmp/rtf_parser_src.jar"/>
<property name="rtf-parser.jar" value="lib/rtf-parser.jar"/>
<available file="${rtf-src.jar}" property="rtf-src.jar.available"/>
<available file="${rtf-parser.jar}" property="rtf-parser.available"/>
<target name="download-rtf-src" unless="rtf-src.jar.available">
<mkdir dir="tmp"/>
<get src="http://www.cobase.cs.ucla.edu/pub/javacc/rtf_parser_src.jar"
dest="tmp/rtf_parser_src.jar"/>
</target>
<target name="build-rtf-parser" depends="download-rtf-src"
unless="rtf-parser.available">
<unjar src="${rtf-src.jar}"
dest="tmp"/>
<javacc target="tmp/rtf/RTFParser.jj"
javacchome="${javacc.home}">
</javacc>
<mkdir dir="tmp/classes"/>
<javac srcdir="tmp" destdir="tmp/classes"/>
<jar destfile="${rtf-parser.jar}" basedir="tmp/classes"/>
<delete>
<fileset dir="tmp">
<exclude name="*.jar"/>
</fileset>
</delete>
</target>
<target name="init-plugin" depends="build-rtf-parser">
<!-- for junit test -->
<mkdir dir="${build.test}/data"/>
<copy file="sample/test.rtf" todir="${build.test}/data"/>
</target>
</project>
>
> Doug
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Nutch-developers mailing list
> Nutch-developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-developers
Re: [Nutch-dev] Re: Licenses
Posted by Doug Cutting <cu...@nutch.org>.
Andy Hedges wrote:
> As you can see for rtf I need to do more than just download the jar
> file as there is not a precompiled version on the net. I have to
> download the source and then run a number of steps to get to the
> resulting jar. I'm not sure that the plugin's build.xml is the place to
> be doing this as it seems to be called twice.
The ant excerpts I sent were careful to only download things once.
http://www.mail-archive.com/nutch-developers%40lists.sourceforge.net/msg03602.html
Conditionals are not straightforward in Ant. First, <available/> is
used to note whether things have already been downloaded. Then an
"unless" attribute is used on the target that actually downloads things
so that the target is conditioned on the results of the <available/>.
Finally, override the "init-plugin" target to depend on the download
task. You could unpack & build the RTF parser in either the download or
the init-plugin task, as you wish.
Does that make sense?
Doug
Re: [Nutch-dev] Re: Licenses
Posted by Andy Hedges <nu...@hedges.net>.
Doug,
As you can see for rtf I need to do more than just download the jar
file as there is not a precompiled version on the net. I have to
download the source and then run a number of steps to get to the
resulting jar. I'm not sure that the plugin's build.xml is the place to
be doing this as it seems to be called twice. Anyway this is just a
first cut to see what you and the rest of the list think we should do in
this situation. Unit tests won't pass as they are dependant on the file
protocol which is dependant on activation...
--- build.properties
javacc.home=/usr/java/javacc
--- build.xml
<project name="parse-rtf" default="jar">
<import file="../build-plugin.xml"/>
<mkdir dir="tmp"/>
<get src="http://www.cobase.cs.ucla.edu/pub/javacc/rtf_parser_src.jar"
dest="tmp/rtf_parser_src.jar"/>
<unjar src="tmp/rtf_parser_src.jar"
dest="tmp"/>
<javacc target="tmp/rtf/RTFParser.jj"
javacchome="${javacc.home}">
</javacc>
<mkdir dir="tmp/classes"/>
<javac srcdir="tmp" destdir="tmp/classes"/>
<jar destfile="lib/rtf-parser.jar" basedir="tmp/classes"/>
<!-- for junit test -->
<mkdir dir="${build.test}/data"/>
<copy file="sample/test.rtf" todir="${build.test}/data"/>
</project>
Doug Cutting wrote:
> Andy Hedges wrote:
>
>> Was just checking through the mailing list to see how things were
>> going with Nutch. It seems that two of my contributions had to be
>> removed because they aren't compatible with the ASF licenses (rtf and
>> mp3 parsers). So what needs to be done to get them back in? Do
>> alternative libraries need to be written/used that are either BSD or
>> ASF licensed? Could you let me know and I'll try and find sometime to
>> rectify the issue.
>
>
> Right, Apache doesn't permit GPL'd stuff on Apache servers. One
> possibility is to rewrite them to use different libraries. Another
> approach is to just get the user to download the required stuff
> themselves.
>
> For example, a plugin's build.xml can download required libraries as
> follows. First, add a few properties noting where the required jar
> lives on the net, and where it should be stored:
>
> <property name="foo.jar" value="lib/my.jar"/>
> <property name="foo.url" value="http://foo.com/foo.jar"/>
>
> http://ant.apache.org/manual/CoreTasks/available.html
>
> Then define a target to download it as needed:
>
> <available file="${foo.jar}" property="foo.jar.available"/>
>
> <target name="download-foo" unless="foo.jar.available">
> <get src="${foo.url}" dest="${foo.jar}"/>
> </target>
>
> http://ant.apache.org/manual/CoreTasks/available.html
> http://ant.apache.org/manual/CoreTasks/get.html
>
> and finally, override the init-plugin task to first download it:
>
> <target name="init-plugin" depends="download-libs"/>
>
> Does this make sense?
>
> Doug
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: 2005 Windows Mobile Application
> Contest
> Submit applications for Windows Mobile(tm)-based Pocket PCs or
> Smartphones
> for the chance to win $25,000 and application distribution. Enter
> today at
> http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click
> _______________________________________________
> Nutch-developers mailing list
> Nutch-developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-developers
Re: Licenses
Posted by Andy Hedges <nu...@hedges.net>.
Yes, makes perfect sense and is almost certainly less work :). Jakarta
Cactus uses a similar methodology that I'm sure we can reuse here. I'll
take a look this weekend and see what I can come up with.
Thanks,
Andy
Doug Cutting wrote:
> Andy Hedges wrote:
>
>> Was just checking through the mailing list to see how things were
>> going with Nutch. It seems that two of my contributions had to be
>> removed because they aren't compatible with the ASF licenses (rtf and
>> mp3 parsers). So what needs to be done to get them back in? Do
>> alternative libraries need to be written/used that are either BSD or
>> ASF licensed? Could you let me know and I'll try and find sometime to
>> rectify the issue.
>
>
> Right, Apache doesn't permit GPL'd stuff on Apache servers. One
> possibility is to rewrite them to use different libraries. Another
> approach is to just get the user to download the required stuff
> themselves.
>
> For example, a plugin's build.xml can download required libraries as
> follows. First, add a few properties noting where the required jar
> lives on the net, and where it should be stored:
>
> <property name="foo.jar" value="lib/my.jar"/>
> <property name="foo.url" value="http://foo.com/foo.jar"/>
>
> http://ant.apache.org/manual/CoreTasks/available.html
>
> Then define a target to download it as needed:
>
> <available file="${foo.jar}" property="foo.jar.available"/>
>
> <target name="download-foo" unless="foo.jar.available">
> <get src="${foo.url}" dest="${foo.jar}"/>
> </target>
>
> http://ant.apache.org/manual/CoreTasks/available.html
> http://ant.apache.org/manual/CoreTasks/get.html
>
> and finally, override the init-plugin task to first download it:
>
> <target name="init-plugin" depends="download-libs"/>
>
> Does this make sense?
>
> Doug