DIH and repeated chunked input

["Norskog, Lance" <la...@divvio.com>]

In http://wiki.apache.org/solr/DataImportHandler there is this
paragraph:
 
    If an API supports chunking (when the dataset is too large) multiple
calls need to be made to complete the process. XPathEntityprocessor
supports this with a transformer. If transformer returns a row which
contains a field $hasMore with a the value "true" the Processor makes
another request with the same url template (The actual value is
recomputed before invoking ). A transformer can pass a totally new url
too for the next call by returning a row which contains a field $nextUrl
whose value must be the complete url for the next call. 
 
 
Does this translate as: "Nobody wrote this yet, but it would be really
cool"?
 
Thanks,
 
Lance

Re: DIH and repeated chunked input

[Shalin Shekhar Mangar <sh...@gmail.com>]

It is implemented. We used this feature to ingest data from a REST API quite
similar to Solr's own.

Our use-case was that the first call to the API returned a token in the xml
response. To get to the next set of results, the value of the token in the
last response needs to be passed as a request parameter to the HTTP API. For
that use-case, simple variable substitution was enough. However if your API
call can't be built with variable substitution, you can write a transformer
yourself which can pass in a completely built URL for the next call.

On Thu, Nov 13, 2008 at 3:55 AM, Norskog, Lance <la...@divvio.com> wrote:

> In http://wiki.apache.org/solr/DataImportHandler there is this
> paragraph:
>
>    If an API supports chunking (when the dataset is too large) multiple
> calls need to be made to complete the process. XPathEntityprocessor
> supports this with a transformer. If transformer returns a row which
> contains a field $hasMore with a the value "true" the Processor makes
> another request with the same url template (The actual value is
> recomputed before invoking ). A transformer can pass a totally new url
> too for the next call by returning a row which contains a field $nextUrl
> whose value must be the complete url for the next call.
>
>
> Does this translate as: "Nobody wrote this yet, but it would be really
> cool"?
>
> Thanks,
>
> Lance
>



-- 
Regards,
Shalin Shekhar Mangar.