You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Tobias Rübner <de...@tobr.eu> on 2011/09/07 10:15:42 UTC
maximum rows for dataimport using solrj
Hi,
I want to trigger the data import handler remotly using the solrj api.
So I added a dih request handler to my solr config and tried to call the
handler as described here
http://wiki.apache.org/solr/SolJava
ModifiableSolrParams params = new ModifiableSolrParams();
params.set("qt", "/dataimport");
params.set("command", "full-import");
params.set("commit", "true");
params.set("debug", "true");
solrServer.query(params);
That worked fine, but it only added the first 10 rows.
So I added a rows param
params.set("rows", 50000000);
which only allows 50M as maximum value.
Having databases with more records, this wouldn't index all entries.
Do you hav any suggestions how to change this?
Thanks,
tob
Re: maximum rows for dataimport using solrj
Posted by Tobias Rübner <de...@tobr.eu>.
Thanks Erik,
but I want to change the whole configuration to e.g. connect to an other
database.
So there is also the need to add/remove (sub-)entities to resolve database
relations.
tob
On Wed, Sep 7, 2011 at 3:04 PM, Erick Erickson <er...@gmail.com>wrote:
> Hossman just posted this, does it help?
>
> You should be able to paramaterize hte table name in the SQL using a
> request parameter...
>
> https://wiki.apache.org/solr/DataImportHandler#Accessing_request_parameters
>
>
> Best
> Erick
>
> On Wed, Sep 7, 2011 at 9:00 AM, Tobias Rübner <de...@tobr.eu> wrote:
> > Thanks for your quick response.
> > The reason why I am using the debug mode is to dynamically add a data
> config
> > with my request.
> > I want to create a new core and add a new configuration to index a new
> > database with the DIH, without changing the solrconfig.xml.
> >
> > Do you see any other options?
> > tob
> >
> >
> >
> >
> >
> > On Wed, Sep 7, 2011 at 12:15 PM, Shalin Shekhar Mangar <
> > shalinmangar@gmail.com> wrote:
> >
> >> On Wed, Sep 7, 2011 at 1:45 PM, Tobias Rübner <de...@tobr.eu> wrote:
> >>
> >> > Hi,
> >> >
> >> > I want to trigger the data import handler remotly using the solrj api.
> >> > So I added a dih request handler to my solr config and tried to call
> the
> >> > handler as described here
> >> > http://wiki.apache.org/solr/SolJava
> >> >
> >> > ModifiableSolrParams params = new ModifiableSolrParams();
> >> > params.set("qt", "/dataimport");
> >> > params.set("command", "full-import");
> >> > params.set("commit", "true");
> >> > params.set("debug", "true");
> >> >
> >> > solrServer.query(params);
> >> >
> >> >
> >> > That worked fine, but it only added the first 10 rows.
> >> > So I added a rows param
> >> > params.set("rows", 50000000);
> >> >
> >> > which only allows 50M as maximum value.
> >> >
> >> > Having databases with more records, this wouldn't index all entries.
> >> >
> >> > Do you hav any suggestions how to change this?
> >> >
> >> >
> >> You are sending debug=true as a request parameter. The debug mode limits
> >> the
> >> number of rows to 10 by default and it also forces indexing to be
> >> synchronous with the request.
> >>
> >> Just remove debug=true from the params and DIH will index all entries.
> >>
> >> --
> >> Regards,
> >> Shalin Shekhar Mangar.
> >>
> >
>
Re: maximum rows for dataimport using solrj
Posted by Erick Erickson <er...@gmail.com>.
Hossman just posted this, does it help?
You should be able to paramaterize hte table name in the SQL using a
request parameter...
https://wiki.apache.org/solr/DataImportHandler#Accessing_request_parameters
Best
Erick
On Wed, Sep 7, 2011 at 9:00 AM, Tobias Rübner <de...@tobr.eu> wrote:
> Thanks for your quick response.
> The reason why I am using the debug mode is to dynamically add a data config
> with my request.
> I want to create a new core and add a new configuration to index a new
> database with the DIH, without changing the solrconfig.xml.
>
> Do you see any other options?
> tob
>
>
>
>
>
> On Wed, Sep 7, 2011 at 12:15 PM, Shalin Shekhar Mangar <
> shalinmangar@gmail.com> wrote:
>
>> On Wed, Sep 7, 2011 at 1:45 PM, Tobias Rübner <de...@tobr.eu> wrote:
>>
>> > Hi,
>> >
>> > I want to trigger the data import handler remotly using the solrj api.
>> > So I added a dih request handler to my solr config and tried to call the
>> > handler as described here
>> > http://wiki.apache.org/solr/SolJava
>> >
>> > ModifiableSolrParams params = new ModifiableSolrParams();
>> > params.set("qt", "/dataimport");
>> > params.set("command", "full-import");
>> > params.set("commit", "true");
>> > params.set("debug", "true");
>> >
>> > solrServer.query(params);
>> >
>> >
>> > That worked fine, but it only added the first 10 rows.
>> > So I added a rows param
>> > params.set("rows", 50000000);
>> >
>> > which only allows 50M as maximum value.
>> >
>> > Having databases with more records, this wouldn't index all entries.
>> >
>> > Do you hav any suggestions how to change this?
>> >
>> >
>> You are sending debug=true as a request parameter. The debug mode limits
>> the
>> number of rows to 10 by default and it also forces indexing to be
>> synchronous with the request.
>>
>> Just remove debug=true from the params and DIH will index all entries.
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
Re: maximum rows for dataimport using solrj
Posted by Tobias Rübner <de...@tobr.eu>.
Thanks for your quick response.
The reason why I am using the debug mode is to dynamically add a data config
with my request.
I want to create a new core and add a new configuration to index a new
database with the DIH, without changing the solrconfig.xml.
Do you see any other options?
tob
On Wed, Sep 7, 2011 at 12:15 PM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
> On Wed, Sep 7, 2011 at 1:45 PM, Tobias Rübner <de...@tobr.eu> wrote:
>
> > Hi,
> >
> > I want to trigger the data import handler remotly using the solrj api.
> > So I added a dih request handler to my solr config and tried to call the
> > handler as described here
> > http://wiki.apache.org/solr/SolJava
> >
> > ModifiableSolrParams params = new ModifiableSolrParams();
> > params.set("qt", "/dataimport");
> > params.set("command", "full-import");
> > params.set("commit", "true");
> > params.set("debug", "true");
> >
> > solrServer.query(params);
> >
> >
> > That worked fine, but it only added the first 10 rows.
> > So I added a rows param
> > params.set("rows", 50000000);
> >
> > which only allows 50M as maximum value.
> >
> > Having databases with more records, this wouldn't index all entries.
> >
> > Do you hav any suggestions how to change this?
> >
> >
> You are sending debug=true as a request parameter. The debug mode limits
> the
> number of rows to 10 by default and it also forces indexing to be
> synchronous with the request.
>
> Just remove debug=true from the params and DIH will index all entries.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: maximum rows for dataimport using solrj
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Wed, Sep 7, 2011 at 1:45 PM, Tobias Rübner <de...@tobr.eu> wrote:
> Hi,
>
> I want to trigger the data import handler remotly using the solrj api.
> So I added a dih request handler to my solr config and tried to call the
> handler as described here
> http://wiki.apache.org/solr/SolJava
>
> ModifiableSolrParams params = new ModifiableSolrParams();
> params.set("qt", "/dataimport");
> params.set("command", "full-import");
> params.set("commit", "true");
> params.set("debug", "true");
>
> solrServer.query(params);
>
>
> That worked fine, but it only added the first 10 rows.
> So I added a rows param
> params.set("rows", 50000000);
>
> which only allows 50M as maximum value.
>
> Having databases with more records, this wouldn't index all entries.
>
> Do you hav any suggestions how to change this?
>
>
You are sending debug=true as a request parameter. The debug mode limits the
number of rows to 10 by default and it also forces indexing to be
synchronous with the request.
Just remove debug=true from the params and DIH will index all entries.
--
Regards,
Shalin Shekhar Mangar.