You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bruno Mannina <bm...@free.fr> on 2014/08/10 01:18:11 UTC
How can I request a big list of values ?
Hi All,
I'm using actually SOLR 3.6 and I have around 91 000 000 docs inside.
All work fine, it's great :)
But now, I would like to request a list of values in the same field
(more than 2000 values)
I know I can use |?q=x:(AAA BBB CCC ...) (my default operator is OR)
but I have a list of 2000 values ! I think it's not the good idea to use
this method.
Can someone help me to find the good solution ?
Can I use a json structure by using a POST method ?
Thanks a lot,
Bruno
|
---
Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active.
http://www.avast.com
Re: How can I request a big list of values ?
Posted by Jack Krupansky <ja...@basetechnology.com>.
Not safe? In what way?
It might be nice to have a specialized SolrJ API for this particular kind of
request, so the API can do the merge. Maybe do it as a class so that you
could have a method that gets invoked as documents trickle back from the
various requests, again so that it is not a massive, blocking request.
-- Jack Krupansky
-----Original Message-----
From: Bruno Mannina
Sent: Sunday, August 10, 2014 6:04 PM
To: solr-user@lucene.apache.org
Subject: Re: How can I request a big list of values ?
Hi Anshum,
I can do it with 3.6 release no ?
my main problem, it's that I have around 2000 values, so I can't use one
request with these values, it's too wide. :'(
I will take a look to generate (like Jack proposes me) several requests,
but even in this case it seems to be not safe...
Le 10/08/2014 19:45, Anshum Gupta a écrit :
> Hi Bruno,
>
> If you would have been on a more recent release,
> https://issues.apache.org/jira/browse/SOLR-6318 would have come in
> handy perhaps.
> You might want to look at patching your version with this though (as a
> work around).
>
> On Sat, Aug 9, 2014 at 4:18 PM, Bruno Mannina <bm...@free.fr> wrote:
>> Hi All,
>>
>> I'm using actually SOLR 3.6 and I have around 91 000 000 docs inside.
>>
>> All work fine, it's great :)
>>
>> But now, I would like to request a list of values in the same field (more
>> than 2000 values)
>>
>> I know I can use |?q=x:(AAA BBB CCC ...) (my default operator is OR)
>>
>> but I have a list of 2000 values ! I think it's not the good idea to use
>> this method.
>>
>> Can someone help me to find the good solution ?
>> Can I use a json structure by using a POST method ?
>>
>> Thanks a lot,
>> Bruno
>> |
>>
>>
>> ---
>> Ce courrier électronique ne contient aucun virus ou logiciel malveillant
>> parce que la protection avast! Antivirus est active.
>> http://www.avast.com
>
>
---
Ce courrier électronique ne contient aucun virus ou logiciel malveillant
parce que la protection avast! Antivirus est active.
http://www.avast.com
Re: How can I request a big list of values ?
Posted by Bruno Mannina <bm...@free.fr>.
Hi Anshum,
I can do it with 3.6 release no ?
my main problem, it's that I have around 2000 values, so I can't use one
request with these values, it's too wide. :'(
I will take a look to generate (like Jack proposes me) several requests,
but even in this case it seems to be not safe...
Le 10/08/2014 19:45, Anshum Gupta a écrit :
> Hi Bruno,
>
> If you would have been on a more recent release,
> https://issues.apache.org/jira/browse/SOLR-6318 would have come in
> handy perhaps.
> You might want to look at patching your version with this though (as a
> work around).
>
> On Sat, Aug 9, 2014 at 4:18 PM, Bruno Mannina <bm...@free.fr> wrote:
>> Hi All,
>>
>> I'm using actually SOLR 3.6 and I have around 91 000 000 docs inside.
>>
>> All work fine, it's great :)
>>
>> But now, I would like to request a list of values in the same field (more
>> than 2000 values)
>>
>> I know I can use |?q=x:(AAA BBB CCC ...) (my default operator is OR)
>>
>> but I have a list of 2000 values ! I think it's not the good idea to use
>> this method.
>>
>> Can someone help me to find the good solution ?
>> Can I use a json structure by using a POST method ?
>>
>> Thanks a lot,
>> Bruno
>> |
>>
>>
>> ---
>> Ce courrier électronique ne contient aucun virus ou logiciel malveillant
>> parce que la protection avast! Antivirus est active.
>> http://www.avast.com
>
>
---
Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active.
http://www.avast.com
Re: How can I request a big list of values ?
Posted by Anshum Gupta <an...@anshumgupta.net>.
Hi Bruno,
If you would have been on a more recent release,
https://issues.apache.org/jira/browse/SOLR-6318 would have come in
handy perhaps.
You might want to look at patching your version with this though (as a
work around).
On Sat, Aug 9, 2014 at 4:18 PM, Bruno Mannina <bm...@free.fr> wrote:
> Hi All,
>
> I'm using actually SOLR 3.6 and I have around 91 000 000 docs inside.
>
> All work fine, it's great :)
>
> But now, I would like to request a list of values in the same field (more
> than 2000 values)
>
> I know I can use |?q=x:(AAA BBB CCC ...) (my default operator is OR)
>
> but I have a list of 2000 values ! I think it's not the good idea to use
> this method.
>
> Can someone help me to find the good solution ?
> Can I use a json structure by using a POST method ?
>
> Thanks a lot,
> Bruno
> |
>
>
> ---
> Ce courrier électronique ne contient aucun virus ou logiciel malveillant
> parce que la protection avast! Antivirus est active.
> http://www.avast.com
--
Anshum Gupta
http://www.anshumgupta.net
Re: How can I request a big list of values ?
Posted by Jack Krupansky <ja...@basetechnology.com>.
The issue is not whether or how to do a massive request, but to recognize
that a single massive request across the network is very clearly an
anti-pattern for modern distributed systems.
Instead of searching for ways to do something "bad", it is better to figure
out how to exploit the positive potential of a system, which in this case is
parallel execution of distributed components.
-- Jack Krupansky
-----Original Message-----
From: Bruno Mannina
Sent: Sunday, August 10, 2014 6:01 PM
To: solr-user@lucene.apache.org
Subject: Re: How can I request a big list of values ?
Hi Jack,
ok but for 2000 values, it means that I must do 40 requests if I choose
to have 50 values by requests :'(
and in my case, user can choose about 8 topics, so it can generate 8
times 40 requests... humm...
is it not possible to send a text, json, xml file ?
Le 10/08/2014 17:38, Jack Krupansky a écrit :
> Generally, "large requests" are an anti-pattern in modern distributed
> systems. Better to have a number of smaller requests executing in parallel
> and then merge the results in the application layer.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Bruno Mannina
> Sent: Saturday, August 9, 2014 7:18 PM
> To: solr-user@lucene.apache.org
> Subject: How can I request a big list of values ?
>
> Hi All,
>
> I'm using actually SOLR 3.6 and I have around 91 000 000 docs inside.
>
> All work fine, it's great :)
>
> But now, I would like to request a list of values in the same field
> (more than 2000 values)
>
> I know I can use |?q=x:(AAA BBB CCC ...) (my default operator is OR)
>
> but I have a list of 2000 values ! I think it's not the good idea to use
> this method.
>
> Can someone help me to find the good solution ?
> Can I use a json structure by using a POST method ?
>
> Thanks a lot,
> Bruno
> |
>
>
> ---
> Ce courrier électronique ne contient aucun virus ou logiciel malveillant
> parce que la protection avast! Antivirus est active.
> http://www.avast.com
>
>
---
Ce courrier électronique ne contient aucun virus ou logiciel malveillant
parce que la protection avast! Antivirus est active.
http://www.avast.com
Re: How can I request a big list of values ?
Posted by Bruno Mannina <bm...@free.fr>.
Hi Jack,
ok but for 2000 values, it means that I must do 40 requests if I choose
to have 50 values by requests :'(
and in my case, user can choose about 8 topics, so it can generate 8
times 40 requests... humm...
is it not possible to send a text, json, xml file ?
Le 10/08/2014 17:38, Jack Krupansky a écrit :
> Generally, "large requests" are an anti-pattern in modern distributed
> systems. Better to have a number of smaller requests executing in
> parallel and then merge the results in the application layer.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Bruno Mannina
> Sent: Saturday, August 9, 2014 7:18 PM
> To: solr-user@lucene.apache.org
> Subject: How can I request a big list of values ?
>
> Hi All,
>
> I'm using actually SOLR 3.6 and I have around 91 000 000 docs inside.
>
> All work fine, it's great :)
>
> But now, I would like to request a list of values in the same field
> (more than 2000 values)
>
> I know I can use |?q=x:(AAA BBB CCC ...) (my default operator is OR)
>
> but I have a list of 2000 values ! I think it's not the good idea to use
> this method.
>
> Can someone help me to find the good solution ?
> Can I use a json structure by using a POST method ?
>
> Thanks a lot,
> Bruno
> |
>
>
> ---
> Ce courrier électronique ne contient aucun virus ou logiciel
> malveillant parce que la protection avast! Antivirus est active.
> http://www.avast.com
>
>
---
Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active.
http://www.avast.com
Re: How can I request a big list of values ?
Posted by Jack Krupansky <ja...@basetechnology.com>.
Generally, "large requests" are an anti-pattern in modern distributed
systems. Better to have a number of smaller requests executing in parallel
and then merge the results in the application layer.
-- Jack Krupansky
-----Original Message-----
From: Bruno Mannina
Sent: Saturday, August 9, 2014 7:18 PM
To: solr-user@lucene.apache.org
Subject: How can I request a big list of values ?
Hi All,
I'm using actually SOLR 3.6 and I have around 91 000 000 docs inside.
All work fine, it's great :)
But now, I would like to request a list of values in the same field
(more than 2000 values)
I know I can use |?q=x:(AAA BBB CCC ...) (my default operator is OR)
but I have a list of 2000 values ! I think it's not the good idea to use
this method.
Can someone help me to find the good solution ?
Can I use a json structure by using a POST method ?
Thanks a lot,
Bruno
|
---
Ce courrier électronique ne contient aucun virus ou logiciel malveillant
parce que la protection avast! Antivirus est active.
http://www.avast.com