You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by liv <li...@hotmail.com> on 2006/12/14 16:16:48 UTC
subcollections
I intend to use nutch with a fairly complex structure of subcollections. I
did some tests and the storage/search performs as expected; however there is
an aspect I may have neglected and cannot find an answer.
How/at which stage are subcollections added to the index structure?
I plan on crawling frequently, adding new sites to existent repository,
merging/reindexing as needed. However if I need to change the subcollection
structure (ie. add a site to a newly created subcollection) I don't want to
recrawl it again. I hope it can be done by simply using the existent/crawled
data.
I tried the following: fetch/index some sites using subcollection1.xml. Then
change to subcollection2.xml. Then reindex - nothing happens in terms of
collections. Same if I do a updatedb/reindex.
Can you please point me something to look at? I am fairly poor in java, and
I hope I won't need to change the source code. Although the need to
"inject" somehow the new names for subcollection field looks rather
peculiar.
Thanks
--
View this message in context: http://www.nabble.com/subcollections-tf2821188.html#a7874307
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: subcollections
Posted by liv <li...@hotmail.com>.
I checked the patch for subcollections
(http://issues.apache.org/jira/browse/NUTCH-201) - although I assumed it is
included in the latest public release 0.8.1.
Compared to the current source code, it looks like having has an extra file
(which doesn't exist in version 0.8.1)
src/plugin/subcollection/src/java/org/apache/nutch/util/DomUtil.java
Could this be the case for "collection" not being updated on re-indexing?
Sami Siren-2 wrote:
>
> liv wrote:
>> - I reindex the db: delete folder "indexes", run the command:
>>
>> bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb crawl/segments/*
>>
>> - then I inspect the resulting db with luke again
>>
>> Unfortunately nothing has changed. Maybe I am missing something... Please
>> tell me if you see anything wrong.
>
> If you did exactly those steps then what happens is that the
> subcollections.xml is read from inside the .job file. You need to
> rebuild the .job to put new file inside of it.
>
> simply do "ant" and rerun indexing and it should work as expected.
>
> --
> Sami Siren
>
>
>
--
View this message in context: http://www.nabble.com/subcollections-tf2821188.html#a7947722
Sent from the Nutch - User mailing list archive at Nabble.com.
A better Drupal (PHP) frontend for OpenSearch RSS
Posted by Robert Douglass <ro...@robshouse.net>.
Hi Nutch folks,
following up on my previous announcement of a Drupal module that
supports OpenRSS search, I'd like to tell you about the OpenSearch RSS
Aggregator[1] that Steven Wittens[2] has written. It lets you set up
multiple feeds which get cached locally and presents them as a unified,
relevance sorted, search result. This lets you integrate multiple Nutch
instances, or integrate Nutch with any other OpenSearch RSS provider.
Furthermore, with the Drupal OpenSearch[3] module included, you can
offer that re-packaged aggregate feed as OpenSearch RSS. Cool! I include
a link to the Digg story[4] to support our efforts to bring more
attention to Nutch, Drupal, and OpenSearch RSS.
cheers,
Robert Douglass
[1] http://drupal.org/project/opensearch_aggregator
[2] http://www.acko.net/blog/drupal-opensearch-aggregator
[3] http://drupal.org/project/opensearch
[4]
http://www.digg.com/software/Drupal_as_a_complete_OpenSearch_processing_suite
Re: subcollections IT DOESN'T WORK!
Posted by liv <li...@hotmail.com>.
look here:
http://issues.apache.org/jira/browse/NUTCH-201?page=all
unfortunately it doesn't work as expected... yet
kauu wrote:
>
> hi , i'm new to nutch ,i want to know what's the useness of the
> subcollection plugin?
> where is the introduction?
>
--
View this message in context: http://www.nabble.com/subcollections-tf2821188.html#a7946767
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: subcollections IT DOESN'T WORK!
Posted by kauu <ba...@gmail.com>.
hi , i'm new to nutch ,i want to know what's the useness of the
subcollection plugin?
where is the introduction?
On 12/19/06, liv <li...@hotmail.com> wrote:
>
>
> I may be loosing all and every credit ... it's still in the same state -
> reindex doesn't change the subcollection field!
>
> I did a REFETCH by mistake (before reindex), and I was happy to notice
> that
> subcollections were changed - but I assumed it happened only due to
> reindex.
>
> However I am looking for REINDEX only - and subcollection field looks that
> it doesn't change (on corresponding changes on subcollection.xml file).
>
> Any help in debugging would be greatly appreciated... however I'm not
> acquinted to java to pursue this by myself.
>
> thanks
>
>
> liv wrote:
> >
> > I have no ideea why this hapened - probably due to luke, because of it
> not
> > re-reading the indexes? very strange!
> >
> > Anyway, it works as it should - after a reindex the subcollection field
> is
> > populated with latest data.
> >
> > Please excuse my insistence and my clumsiness, and thanks for your
> > answers.
> >
> >
> >
> > liv wrote:
> >>
> >> Unfortunately my java knowledge is too poor to debug this one. However
> I
> >> doubt that the file "subcollections.xml" from inside the nutch-xxx.jobis
> >> used. This because the file nutchxxx.job is old enough - has the date
> >> since the day I made he nutch installation.
> >>
> >>
> >> Sami Siren-2 wrote:
> >>>
> >>> liv wrote:
> >>>> - I reindex the db: delete folder "indexes", run the command:
> >>>>
> >>>> bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb
> >>>> crawl/segments/*
> >>>>
> >>>> - then I inspect the resulting db with luke again
> >>>>
> >>>> Unfortunately nothing has changed. Maybe I am missing something...
> >>>> Please
> >>>> tell me if you see anything wrong.
> >>>
> >>> If you did exactly those steps then what happens is that the
> >>> subcollections.xml is read from inside the .job file. You need to
> >>> rebuild the .job to put new file inside of it.
> >>>
> >>> simply do "ant" and rerun indexing and it should work as expected.
> >>>
> >>> --
> >>> Sami Siren
> >>>
> >>>
> >>>
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/subcollections-tf2821188.html#a7935139
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>
--
www.babatu.com
Re: subcollections IT DOESN'T WORK!
Posted by liv <li...@hotmail.com>.
I may be loosing all and every credit ... it's still in the same state -
reindex doesn't change the subcollection field!
I did a REFETCH by mistake (before reindex), and I was happy to notice that
subcollections were changed - but I assumed it happened only due to reindex.
However I am looking for REINDEX only - and subcollection field looks that
it doesn't change (on corresponding changes on subcollection.xml file).
Any help in debugging would be greatly appreciated... however I'm not
acquinted to java to pursue this by myself.
thanks
liv wrote:
>
> I have no ideea why this hapened - probably due to luke, because of it not
> re-reading the indexes? very strange!
>
> Anyway, it works as it should - after a reindex the subcollection field is
> populated with latest data.
>
> Please excuse my insistence and my clumsiness, and thanks for your
> answers.
>
>
>
> liv wrote:
>>
>> Unfortunately my java knowledge is too poor to debug this one. However I
>> doubt that the file "subcollections.xml" from inside the nutch-xxx.job is
>> used. This because the file nutchxxx.job is old enough - has the date
>> since the day I made he nutch installation.
>>
>>
>> Sami Siren-2 wrote:
>>>
>>> liv wrote:
>>>> - I reindex the db: delete folder "indexes", run the command:
>>>>
>>>> bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb
>>>> crawl/segments/*
>>>>
>>>> - then I inspect the resulting db with luke again
>>>>
>>>> Unfortunately nothing has changed. Maybe I am missing something...
>>>> Please
>>>> tell me if you see anything wrong.
>>>
>>> If you did exactly those steps then what happens is that the
>>> subcollections.xml is read from inside the .job file. You need to
>>> rebuild the .job to put new file inside of it.
>>>
>>> simply do "ant" and rerun indexing and it should work as expected.
>>>
>>> --
>>> Sami Siren
>>>
>>>
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/subcollections-tf2821188.html#a7935139
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: subcollections IT WORKS
Posted by WebDev Freak <we...@gmail.com>.
Were you able by any chance figure out how to search on multiple
subcollections? For example let's say you have the following
subcollections: books, magazines, cd, dvd, software. I would like to have a
search page with checkboxes to pick any of the subcollections. For example
search on books, magazines and cd's or just search in dvd and software.
Etc. Does anybody know how to do this? Thanks.
On 12/18/06, liv <li...@hotmail.com> wrote:
>
>
> I have no ideea why this hapened - probably due to luke, because of it not
> re-reading the indexes? very strange!
>
> Anyway, it works as it should - after a reindex the subcollection field is
> populated with latest data.
>
> Please excuse my insistence and my clumsiness, and thanks for your
> answers.
>
>
>
> liv wrote:
> >
> > Unfortunately my java knowledge is too poor to debug this one. However I
> > doubt that the file "subcollections.xml" from inside the nutch-xxx.jobis
> > used. This because the file nutchxxx.job is old enough - has the date
> > since the day I made he nutch installation.
> >
> >
> > Sami Siren-2 wrote:
> >>
> >> liv wrote:
> >>> - I reindex the db: delete folder "indexes", run the command:
> >>>
> >>> bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb
> >>> crawl/segments/*
> >>>
> >>> - then I inspect the resulting db with luke again
> >>>
> >>> Unfortunately nothing has changed. Maybe I am missing something...
> >>> Please
> >>> tell me if you see anything wrong.
> >>
> >> If you did exactly those steps then what happens is that the
> >> subcollections.xml is read from inside the .job file. You need to
> >> rebuild the .job to put new file inside of it.
> >>
> >> simply do "ant" and rerun indexing and it should work as expected.
> >>
> >> --
> >> Sami Siren
> >>
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/subcollections-tf2821188.html#a7930248
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>
Re: subcollections IT WORKS
Posted by liv <li...@hotmail.com>.
I have no ideea why this hapened - probably due to luke, because of it not
re-reading the indexes? very strange!
Anyway, it works as it should - after a reindex the subcollection field is
populated with latest data.
Please excuse my insistence and my clumsiness, and thanks for your answers.
liv wrote:
>
> Unfortunately my java knowledge is too poor to debug this one. However I
> doubt that the file "subcollections.xml" from inside the nutch-xxx.job is
> used. This because the file nutchxxx.job is old enough - has the date
> since the day I made he nutch installation.
>
>
> Sami Siren-2 wrote:
>>
>> liv wrote:
>>> - I reindex the db: delete folder "indexes", run the command:
>>>
>>> bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb
>>> crawl/segments/*
>>>
>>> - then I inspect the resulting db with luke again
>>>
>>> Unfortunately nothing has changed. Maybe I am missing something...
>>> Please
>>> tell me if you see anything wrong.
>>
>> If you did exactly those steps then what happens is that the
>> subcollections.xml is read from inside the .job file. You need to
>> rebuild the .job to put new file inside of it.
>>
>> simply do "ant" and rerun indexing and it should work as expected.
>>
>> --
>> Sami Siren
>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/subcollections-tf2821188.html#a7930248
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: subcollections
Posted by liv <li...@hotmail.com>.
Unfortunately my java knowledge is too poor to debug this one. However I
doubt that the file "subcollections.xml" from inside the nutch-xxx.job is
used. This because the file nutchxxx.job is old enough - has the date since
the day I made he nutch installation.
Initially I use file "subcollections.xml" from inside folder "conf". I add2
collections named "test1", "test2", I fetch/index, then I inspect the db
with luke - and i see the values "test1" and "test2".
Then I change the same file "subcollections.xml", rename collections to
"new1", "new2", reindex and inspect the contents. I see the same values
"test1","test2".
If the file from inside .job would have been used, then I shouldn't see the
initial collections... not the ones I defined at step 1.
Maybe you point me to another .job file... Or maybe the subcollections.xml
is not re-parsed when indexing... unfortunately I cannot trace/debug it.
Sami Siren-2 wrote:
>
> liv wrote:
>> - I reindex the db: delete folder "indexes", run the command:
>>
>> bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb crawl/segments/*
>>
>> - then I inspect the resulting db with luke again
>>
>> Unfortunately nothing has changed. Maybe I am missing something... Please
>> tell me if you see anything wrong.
>
> If you did exactly those steps then what happens is that the
> subcollections.xml is read from inside the .job file. You need to
> rebuild the .job to put new file inside of it.
>
> simply do "ant" and rerun indexing and it should work as expected.
>
> --
> Sami Siren
>
>
>
--
View this message in context: http://www.nabble.com/subcollections-tf2821188.html#a7929866
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: subcollections
Posted by Sami Siren <ss...@gmail.com>.
liv wrote:
> - I reindex the db: delete folder "indexes", run the command:
>
> bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb crawl/segments/*
>
> - then I inspect the resulting db with luke again
>
> Unfortunately nothing has changed. Maybe I am missing something... Please
> tell me if you see anything wrong.
If you did exactly those steps then what happens is that the
subcollections.xml is read from inside the .job file. You need to
rebuild the .job to put new file inside of it.
simply do "ant" and rerun indexing and it should work as expected.
--
Sami Siren
Re: subcollections
Posted by liv <li...@hotmail.com>.
Thanks for your quick answer. However I tried this several times and it
doesn't work. Here is how I do:
- I fetch/index 2 sites, having subcolelctions.xml file as follws:
<?xml version="1.0" encoding="UTF-8"?>
<subcollections>
<subcollection>
<name>test1</name>
<id>test1</id>
<whitelist>
http://npo.d1.com/
</whitelist>
<blacklist />
</subcollection>
<subcollection>
<name>test2</name>
<id>test2</id>
<whitelist>
http://rollout.d1.com/
</whitelist>
<blacklist />
</subcollection>
<subcollection>
<name>test3</name>
<id>test3</id>
<whitelist>
http://npo.d1.com/
http://rollout.d1.com/
</whitelist>
<blacklist />
</subcollection>
</subcollections>
- I inspect the resulting db with luke; the subcollection field is populated
with " test1 test3" and " test2 test3" values respectively
- I change the names of collections in the file subcollections.xml as
follows:
<?xml version="1.0" encoding="UTF-8"?>
<subcollections>
<subcollection>
<name>new1</name>
<id>new1</id>
<whitelist>
http://npo.d1.com/
</whitelist>
<blacklist />
</subcollection>
<subcollection>
<name>new2</name>
<id>new2</id>
<whitelist>
http://rollout.d1.com/
</whitelist>
<blacklist />
</subcollection>
<subcollection>
<name>new3</name>
<id>new3</id>
<whitelist>
http://npo.d1.com/
http://rollout.d1.com/
</whitelist>
<blacklist />
</subcollection>
</subcollections>
- I reindex the db: delete folder "indexes", run the command:
bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb crawl/segments/*
- then I inspect the resulting db with luke again
Unfortunately nothing has changed. Maybe I am missing something... Please
tell me if you see anything wrong.
Thanks!
Sami Siren-2 wrote:
>
> liv wrote:
>> I intend to use nutch with a fairly complex structure of subcollections.
>> I
>> did some tests and the storage/search performs as expected; however there
>> is
>> an aspect I may have neglected and cannot find an answer.
>>
>> How/at which stage are subcollections added to the index structure?
>
> If you are talking about the subcollections generated by the
> subcollection plugin then the subcollection data is stored at indexing
> phase.
>
>> I plan on crawling frequently, adding new sites to existent repository,
>> merging/reindexing as needed. However if I need to change the
>> subcollection
>> structure (ie. add a site to a newly created subcollection) I don't want
>> to
>> recrawl it again. I hope it can be done by simply using the
>> existent/crawled
>> data.
>
> no need to recrawl, unfortunately you still need to reindex.
>
> --
> Sami Siren
>
>
--
View this message in context: http://www.nabble.com/subcollections-tf2821188.html#a7889945
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: subcollections
Posted by Sami Siren <ss...@gmail.com>.
liv wrote:
> I intend to use nutch with a fairly complex structure of subcollections. I
> did some tests and the storage/search performs as expected; however there is
> an aspect I may have neglected and cannot find an answer.
>
> How/at which stage are subcollections added to the index structure?
If you are talking about the subcollections generated by the
subcollection plugin then the subcollection data is stored at indexing
phase.
> I plan on crawling frequently, adding new sites to existent repository,
> merging/reindexing as needed. However if I need to change the subcollection
> structure (ie. add a site to a newly created subcollection) I don't want to
> recrawl it again. I hope it can be done by simply using the existent/crawled
> data.
no need to recrawl, unfortunately you still need to reindex.
--
Sami Siren