Apache Nutch Crawling Dynamic Products on Pages

[Kuldeep Sharma <ku...@hcl.com>]

Hi All,

We have a page that contains some dynamic products. So when I am crawling site, that page gets crawled but when searching for that particular product, its showing no results.
Can you guys help me, How can I crawl these dynamic product on page? Is there any way to achieve this and also can we use sitemaps with Nutch?

---------------------------
Regards,
Kuldeep Sharma



::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.

----------------------------------------------------------------------------------------------------------------------------------------------------

Re: Apache Nutch Crawling Dynamic Products on Pages

[Chris Hostetter <ho...@fucit.org>]

: We have a page that contains some dynamic products. So when I am 
: crawling site, that page gets crawled but when searching for that 
: particular product, its showing no results. Can you guys help me, How 
: can I crawl these dynamic product on page? Is there any way to achieve 
: this and also can we use sitemaps with Nutch?

it's not clear wether your problem is that nutch isn't crawling the pages 
that you want because of the dynamic URLs, or if nutch is crawling the 
pages that you want, but then you are having trouble searching and finding 
those pages because of how your (Solr?) index is configured.

either way: i would suggest you start by addressing your question to the 
nutch user list -- but you almost certainly need to provide more details 
about the types or urls, types of content, how you are running nutch's 
crawl command, logs you see, results you see when executing searches 
etc...

https://nutch.apache.org/mailing_lists.html#Users

In the event that you confirm nutch is crawling the documents and putting 
them -- including all the data you expect -- into a solr index, but they 
are not being returned by the search terms you expect, then please 
start a new thread on the solr-user@lucene mailing list with details about 
your solr configuration, the documents in your index (the specifics of 
how/where nutch got those documents probably isn't relevant on solr-user) 
and what results you get from your query...

https://wiki.apache.org/solr/UsingMailingLists


-Hoss