You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by B B <bb...@gmail.com> on 2011/09/16 18:59:00 UTC

DIH URLDataSource paginating with $nextUrl & $hasMore

Has anyone successfully setup DIH URLDataSource to paginate imports
using  $nextUrl & $hasMore ?:

http://wiki.apache.org/solr/DataImportHandler#Configuration_in_data-config.xml-1

It's not working for me, the data imports on the first page, but no
subsequent calls to the data source are processed.  Here is the
excerpt from my data-config.xml:
<field column="$hasMore" xpath="/results/page/has_more" />
<field column="$nextUrl" xpath="/results/page/next_url" />

The response from the data source has:
<results>
   <pagination>
       <has_more>true</has_more>
       <next_url>http://host/foo/import.xml?start=2</next_url>
   </pagination>
   <docs>
   ...
   </docs>
</results>

Re: DIH URLDataSource paginating with $nextUrl & $hasMore

Posted by B B <bb...@gmail.com>.

BTW - ignore the xpath typo, it should read:

 <field column="$hasMore" xpath="/results/pagination/has_more" />
 <field column="$nextUrl" xpath="/results/pagination/next_url" />



On Fri, Sep 16, 2011 at 12:59 PM, B B <bb...@gmail.com> wrote:
> Has anyone successfully setup DIH URLDataSource to paginate imports
> using  $nextUrl & $hasMore ?:
>
> http://wiki.apache.org/solr/DataImportHandler#Configuration_in_data-config.xml-1
>
> It's not working for me, the data imports on the first page, but no
> subsequent calls to the data source are processed.  Here is the
> excerpt from my data-config.xml:
> <field column="$hasMore" xpath="/results/page/has_more" />
> <field column="$nextUrl" xpath="/results/page/next_url" />
>
> The response from the data source has:
> <results>
>    <pagination>
>        <has_more>true</has_more>
>        <next_url>http://host/foo/import.xml?start=2</next_url>
>    </pagination>
>    <docs>
>    ...
>    </docs>
> </results>
>