You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Fred Zimmerman <wf...@nimblebooks.com> on 2011/09/23 01:44:59 UTC
strategy for post-processing answer set
>
> Hi,
I would like to take the HTML documents that are the result of a Solr search
and combine them into a single HTML document that combines the body text of
each individual document. What is a good strategy for this? I am crawling
with Nutch and Carrot2 for clustering.
Fred
Re: strategy for post-processing answer set
Posted by Fred Zimmerman <wf...@nimblebooks.com>.
can you say a bit more about this? I see Velocity and will download it and
start playing around but I am not quite sure I understand all the steps that
you are suggesting. Fred
On Thu, Sep 22, 2011 at 19:51, Markus Jelsma <ma...@openindex.io>wrote:
> Hi,
>
> Solr support the Velocity template engine and has veyr good support. Ideal
> for
> generating properly formatted output from the search engine. There's a
> clustering example and it's easy to format documents indexed by Nutch.
>
> http://wiki.apache.org/solr/VelocityResponseWriter
>
> Cheers
>
> > > Hi,
> >
> > I would like to take the HTML documents that are the result of a Solr
> > search and combine them into a single HTML document that combines the
> body
> > text of each individual document. What is a good strategy for this? I am
> > crawling with Nutch and Carrot2 for clustering.
> > Fred
>
Re: strategy for post-processing answer set
Posted by Chris Hostetter <ho...@fucit.org>.
: it looks to me as if Solr just brings back the URLs. what I want to do is to
: get the actual documents in the answer set, simplify their HTML and remove
: all the javascript, ads, etc., and append them into a single document.
:
: Now ... does Nutch already have the documents? can I get them from its db?
: or do I have to go get the documents again with something like a wget?
i *think* what you are saying is that:
a) you built your index using nutch
b) when you query Solr, you only get back a "url" field for each matching
document
c) what you want is to combine the whole text of webpages corrisponding to
all of those urls into one massive html page
If that's the case,then you should either:
1) ask on the nutch-user mailing list about how to "store" the whole
content of web pages that nutch crawls so you cna build up a page like
this (nutch may already be doing it, i don't know -- depends on the
schema)
2) write custom client code (probably outside of the score of velocity) to
re-fetch these urls "at query time" and parse them and combine them as you
see fit.
which approach is right for you all depends on your goals and use case --
but solr can only give you back the fields you store in it.
-Hoss
Re: strategy for post-processing answer set
Posted by Fred Zimmerman <wf...@nimblebooks.com>.
ok. this is a very basic question so please bear with me.
I see where the velocity templates are and I have looked at the
documentation and get the idea of how to write them.
it looks to me as if Solr just brings back the URLs. what I want to do is to
get the actual documents in the answer set, simplify their HTML and remove
all the javascript, ads, etc., and append them into a single document.
Now ... does Nutch already have the documents? can I get them from its db?
or do I have to go get the documents again with something like a wget?
Fred
On Fri, Sep 23, 2011 at 16:02, Erik Hatcher <er...@gmail.com> wrote:
> conf/velocity by default. See Solr's example configuration.
>
> Erik
>
> On Sep 23, 2011, at 12:37, Fred Zimmerman <wf...@nimblebooks.com> wrote:
>
> > ok, answered my own question, found velocity rw in solrconfig.xml. next
> > question:
> >
> > where does velocity look for its templates?
> >
> > -----------------------------------------------------
> > Subscribe to the Nimble Books Mailing List http://eepurl.com/czS- for
> > monthly updates
> >
> >
> >
> > On Fri, Sep 23, 2011 at 11:57, Fred Zimmerman <wf...@nimblebooks.com>
> wrote:
> >
> >> This seems to be out of date. I am running Solr 3.4
> >>
> >> * the file structure of apachehome/contrib is different and I don't see
> >> velocity anywhere underneath
> >> * the page referenced below only talks about Solr 1.4 and 4.0
> >>
> >> ?
> >>
> >> On Thu, Sep 22, 2011 at 19:51, Markus Jelsma <
> markus.jelsma@openindex.io>wrote:
> >>
> >>> Hi,
> >>>
> >>> Solr support the Velocity template engine and has veyr good support.
> Ideal
> >>> for
> >>> generating properly formatted output from the search engine. There's a
> >>> clustering example and it's easy to format documents indexed by Nutch.
> >>>
> >>> http://wiki.apache.org/solr/VelocityResponseWriter
> >>>
> >>> Cheers
> >>>
> >>>>> Hi,
> >>>>
> >>>> I would like to take the HTML documents that are the result of a Solr
> >>>> search and combine them into a single HTML document that combines the
> >>> body
> >>>> text of each individual document. What is a good strategy for this? I
> >>> am
> >>>> crawling with Nutch and Carrot2 for clustering.
> >>>> Fred
> >>>
> >>
> >>
>
Re: strategy for post-processing answer set
Posted by Erik Hatcher <er...@gmail.com>.
conf/velocity by default. See Solr's example configuration.
Erik
On Sep 23, 2011, at 12:37, Fred Zimmerman <wf...@nimblebooks.com> wrote:
> ok, answered my own question, found velocity rw in solrconfig.xml. next
> question:
>
> where does velocity look for its templates?
>
> -----------------------------------------------------
> Subscribe to the Nimble Books Mailing List http://eepurl.com/czS- for
> monthly updates
>
>
>
> On Fri, Sep 23, 2011 at 11:57, Fred Zimmerman <wf...@nimblebooks.com> wrote:
>
>> This seems to be out of date. I am running Solr 3.4
>>
>> * the file structure of apachehome/contrib is different and I don't see
>> velocity anywhere underneath
>> * the page referenced below only talks about Solr 1.4 and 4.0
>>
>> ?
>>
>> On Thu, Sep 22, 2011 at 19:51, Markus Jelsma <ma...@openindex.io>wrote:
>>
>>> Hi,
>>>
>>> Solr support the Velocity template engine and has veyr good support. Ideal
>>> for
>>> generating properly formatted output from the search engine. There's a
>>> clustering example and it's easy to format documents indexed by Nutch.
>>>
>>> http://wiki.apache.org/solr/VelocityResponseWriter
>>>
>>> Cheers
>>>
>>>>> Hi,
>>>>
>>>> I would like to take the HTML documents that are the result of a Solr
>>>> search and combine them into a single HTML document that combines the
>>> body
>>>> text of each individual document. What is a good strategy for this? I
>>> am
>>>> crawling with Nutch and Carrot2 for clustering.
>>>> Fred
>>>
>>
>>
Re: strategy for post-processing answer set
Posted by Fred Zimmerman <wf...@nimblebooks.com>.
ok, answered my own question, found velocity rw in solrconfig.xml. next
question:
where does velocity look for its templates?
-----------------------------------------------------
Subscribe to the Nimble Books Mailing List http://eepurl.com/czS- for
monthly updates
On Fri, Sep 23, 2011 at 11:57, Fred Zimmerman <wf...@nimblebooks.com> wrote:
> This seems to be out of date. I am running Solr 3.4
>
> * the file structure of apachehome/contrib is different and I don't see
> velocity anywhere underneath
> * the page referenced below only talks about Solr 1.4 and 4.0
>
> ?
>
> On Thu, Sep 22, 2011 at 19:51, Markus Jelsma <ma...@openindex.io>wrote:
>
>> Hi,
>>
>> Solr support the Velocity template engine and has veyr good support. Ideal
>> for
>> generating properly formatted output from the search engine. There's a
>> clustering example and it's easy to format documents indexed by Nutch.
>>
>> http://wiki.apache.org/solr/VelocityResponseWriter
>>
>> Cheers
>>
>> > > Hi,
>> >
>> > I would like to take the HTML documents that are the result of a Solr
>> > search and combine them into a single HTML document that combines the
>> body
>> > text of each individual document. What is a good strategy for this? I
>> am
>> > crawling with Nutch and Carrot2 for clustering.
>> > Fred
>>
>
>
Re: strategy for post-processing answer set
Posted by Fred Zimmerman <wf...@nimblebooks.com>.
This seems to be out of date. I am running Solr 3.4
* the file structure of apachehome/contrib is different and I don't see
velocity anywhere underneath
* the page referenced below only talks about Solr 1.4 and 4.0
?
On Thu, Sep 22, 2011 at 19:51, Markus Jelsma <ma...@openindex.io>wrote:
> Hi,
>
> Solr support the Velocity template engine and has veyr good support. Ideal
> for
> generating properly formatted output from the search engine. There's a
> clustering example and it's easy to format documents indexed by Nutch.
>
> http://wiki.apache.org/solr/VelocityResponseWriter
>
> Cheers
>
> > > Hi,
> >
> > I would like to take the HTML documents that are the result of a Solr
> > search and combine them into a single HTML document that combines the
> body
> > text of each individual document. What is a good strategy for this? I am
> > crawling with Nutch and Carrot2 for clustering.
> > Fred
>
Re: strategy for post-processing answer set
Posted by Markus Jelsma <ma...@openindex.io>.
Hi,
Solr support the Velocity template engine and has veyr good support. Ideal for
generating properly formatted output from the search engine. There's a
clustering example and it's easy to format documents indexed by Nutch.
http://wiki.apache.org/solr/VelocityResponseWriter
Cheers
> > Hi,
>
> I would like to take the HTML documents that are the result of a Solr
> search and combine them into a single HTML document that combines the body
> text of each individual document. What is a good strategy for this? I am
> crawling with Nutch and Carrot2 for clustering.
> Fred