You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Bauer, Herbert S. (Scott)" <Ba...@mayo.edu> on 2016/03/18 15:02:07 UTC
Serializing Queries
Has anyone in this group solved the problem of serializing complex boolean queries (Some of our clauses have span and other query types)? Our Java RMI depends upon being able to do so. I have seen posts that say you can just parse the string representation but apparently that only works on simple query representations. I’m looking at the CoreParser and it’s supporting xml parsing capabilities with an eye toward Marshalling the boolean query into a DOM object and unmarshalling it on the server side using some of the support implied by the CoreParser and related classes. -scott
Re: Serializing Queries
Posted by "Bauer, Herbert S. (Scott)" <Ba...@mayo.edu>.
This seems to work for me. Unfortunately I had to stick with the 2.2.1
version myself. Thanks to our use of an old version of spring I had some
asm version clashes. I did also make use of the custom serializers found
here:
https://github.com/magro/kryo-serializers.git
writing:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
Output output = new Output(baos);
Kryo kryo = new Kryo();
kryo.setRegistrationRequired(false);
kryo.setInstantiatorStrategy(new StdInstantiatorStrategy());
UnmodifiableCollectionsSerializer.registerSerializers(kryo);
kryo.register(Arrays.asList("").getClass(), new
ArraysAsListSerializer());
kryo.register(BooleanClause.class);
kryo.register(Query.class);
kryo.register(Occur.class);
kryo.register(ToParentBlockJoinQuery.class);
kryo.register(QueryBitSetProducer.class);
SynchronizedCollectionsSerializer.registerSerializers(kryo);
kryo.writeClassAndObject(output, (List<BooleanClause>)
builder.build().clauses());
output.close();
String outputString =
Base64.encodeBase64String(baos.toByteArray());
Reading:
String inputString = (String) in.readObject();
ByteArrayInputStream bais = new
ByteArrayInputStream(Base64.decodeBase64(inputString));
Input input = new Input(bais);
Kryo kryo = new Kryo();
kryo.setRegistrationRequired(false);
kryo.setInstantiatorStrategy(new StdInstantiatorStrategy());
UnmodifiableCollectionsSerializer.registerSerializers(kryo);
kryo.register(Arrays.asList("").getClass(), new
ArraysAsListSerializer());
kryo.register(BooleanClause.class);
kryo.register(Query.class);
kryo.register(Occur.class);
kryo.register(ToParentBlockJoinQuery.class);
kryo.register(QueryBitSetProducer.class);
SynchronizedCollectionsSerializer.registerSerializers(kryo);
@SuppressWarnings("unchecked")
List<BooleanClause> queryObject = (List<BooleanClause>)
kryo.readClassAndObject(input);
input.close();
builder = new BooleanQuery.Builder();
for (BooleanClause clause : queryObject) {
builder.add(clause);
}
Base64 didn¹t line up perfectly either, but I found an apache commons
version that worked. Not sure all the class registration was necessary.
I¹m really grateful for the code sample and direction. This seems to be a
very effective tool for solving a lot of serialization problems that could
have been tough to solve.
On 3/18/16, 10:44 AM, "McKinley, James T" <ja...@cengage.com>
wrote:
>We use Kryo to pass query objects between hosts:
>
>https://github.com/EsotericSoftware/kryo
>
>We initially had some trouble with it creating dynamic classes and
>running out of PermGen space but we got around that using an ObjectPool:
>
>http://commons.apache.org/proper/commons-pool/api-1.6/org/apache/commons/p
>ool/impl/StackObjectPool.html
>
>I've not looked at the project recently, we're using version 2.21 and
>there's a 3.0.0 now, so they may have solved the issues we had and made
>things nicer, but here's how we're doing it with 2.21:
>
>To serialize:
>
>static private ObjectPool<Kryo> pool = new StackObjectPool<Kryo>(new
>KryoFactory(), 75, 75);
>
>ByteArrayOutputStream baos = new ByteArrayOutputStream();
>Output output = new Output(baos);
>kryo = pool.borrowObject();
>kryo.writeClassAndObject(output, query);
>pool.returnObject(kryo);
>output.close();
>String base64EncodedSerializedObject =
>Base64.encodeBytes(baos.toByteArray());
>
>Where query is a Lucene Query object (I've left out the error handling
>for brevity).
>
>To deserialize:
>
>ByteArrayInputStream bais = new
>ByteArrayInputStream(Base64.decode(encodedQuery));
>Input input = new Input(bais);
>kryo = pool.borrowObject();
>deserializedQueryObject = (Query) kryo.readClassAndObject(input);
>pool.returnObject(kryo);
>input.close();
>
>Hope that might help.
>
>Jim
>
>________________________________________
>From: Bauer, Herbert S. (Scott) <Ba...@mayo.edu>
>Sent: 18 March 2016 10:02
>To: java-user@lucene.apache.org
>Subject: Serializing Queries
>
>Has anyone in this group solved the problem of serializing complex
>boolean queries (Some of our clauses have span and other query types)?
>Our Java RMI depends upon being able to do so. I have seen posts that
>say you can just parse the string representation but apparently that only
>works on simple query representations. I¹m looking at the CoreParser
>and it¹s supporting xml parsing capabilities with an eye toward
>Marshalling the boolean query into a DOM object and unmarshalling it on
>the server side using some of the support implied by the CoreParser and
>related classes. -scott
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Serializing Queries
Posted by "Bauer, Herbert S. (Scott)" <Ba...@mayo.edu>.
Thanks James:
This looks promising. I¹ll repost when I¹ve had a chance to implement
this. -scott
On 3/18/16, 10:44 AM, "McKinley, James T" <ja...@cengage.com>
wrote:
>We use Kryo to pass query objects between hosts:
>
>https://github.com/EsotericSoftware/kryo
>
>We initially had some trouble with it creating dynamic classes and
>running out of PermGen space but we got around that using an ObjectPool:
>
>http://commons.apache.org/proper/commons-pool/api-1.6/org/apache/commons/p
>ool/impl/StackObjectPool.html
>
>I've not looked at the project recently, we're using version 2.21 and
>there's a 3.0.0 now, so they may have solved the issues we had and made
>things nicer, but here's how we're doing it with 2.21:
>
>To serialize:
>
>static private ObjectPool<Kryo> pool = new StackObjectPool<Kryo>(new
>KryoFactory(), 75, 75);
>
>ByteArrayOutputStream baos = new ByteArrayOutputStream();
>Output output = new Output(baos);
>kryo = pool.borrowObject();
>kryo.writeClassAndObject(output, query);
>pool.returnObject(kryo);
>output.close();
>String base64EncodedSerializedObject =
>Base64.encodeBytes(baos.toByteArray());
>
>Where query is a Lucene Query object (I've left out the error handling
>for brevity).
>
>To deserialize:
>
>ByteArrayInputStream bais = new
>ByteArrayInputStream(Base64.decode(encodedQuery));
>Input input = new Input(bais);
>kryo = pool.borrowObject();
>deserializedQueryObject = (Query) kryo.readClassAndObject(input);
>pool.returnObject(kryo);
>input.close();
>
>Hope that might help.
>
>Jim
>
>________________________________________
>From: Bauer, Herbert S. (Scott) <Ba...@mayo.edu>
>Sent: 18 March 2016 10:02
>To: java-user@lucene.apache.org
>Subject: Serializing Queries
>
>Has anyone in this group solved the problem of serializing complex
>boolean queries (Some of our clauses have span and other query types)?
>Our Java RMI depends upon being able to do so. I have seen posts that
>say you can just parse the string representation but apparently that only
>works on simple query representations. I¹m looking at the CoreParser
>and it¹s supporting xml parsing capabilities with an eye toward
>Marshalling the boolean query into a DOM object and unmarshalling it on
>the server side using some of the support implied by the CoreParser and
>related classes. -scott
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Serializing Queries
Posted by "McKinley, James T" <ja...@cengage.com>.
We use Kryo to pass query objects between hosts:
https://github.com/EsotericSoftware/kryo
We initially had some trouble with it creating dynamic classes and running out of PermGen space but we got around that using an ObjectPool:
http://commons.apache.org/proper/commons-pool/api-1.6/org/apache/commons/pool/impl/StackObjectPool.html
I've not looked at the project recently, we're using version 2.21 and there's a 3.0.0 now, so they may have solved the issues we had and made things nicer, but here's how we're doing it with 2.21:
To serialize:
static private ObjectPool<Kryo> pool = new StackObjectPool<Kryo>(new KryoFactory(), 75, 75);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
Output output = new Output(baos);
kryo = pool.borrowObject();
kryo.writeClassAndObject(output, query);
pool.returnObject(kryo);
output.close();
String base64EncodedSerializedObject = Base64.encodeBytes(baos.toByteArray());
Where query is a Lucene Query object (I've left out the error handling for brevity).
To deserialize:
ByteArrayInputStream bais = new ByteArrayInputStream(Base64.decode(encodedQuery));
Input input = new Input(bais);
kryo = pool.borrowObject();
deserializedQueryObject = (Query) kryo.readClassAndObject(input);
pool.returnObject(kryo);
input.close();
Hope that might help.
Jim
________________________________________
From: Bauer, Herbert S. (Scott) <Ba...@mayo.edu>
Sent: 18 March 2016 10:02
To: java-user@lucene.apache.org
Subject: Serializing Queries
Has anyone in this group solved the problem of serializing complex boolean queries (Some of our clauses have span and other query types)? Our Java RMI depends upon being able to do so. I have seen posts that say you can just parse the string representation but apparently that only works on simple query representations. I’m looking at the CoreParser and it’s supporting xml parsing capabilities with an eye toward Marshalling the boolean query into a DOM object and unmarshalling it on the server side using some of the support implied by the CoreParser and related classes. -scott
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Serializing Queries
Posted by "Bauer, Herbert S. (Scott)" <Ba...@mayo.edu>.
Thanks Ahmet:
I¹ve seen these, but I don¹t find any mechanism here helping to get me
from the query object to the xml or a DOM object. -scott
On 3/18/16, 11:35 AM, "Ahmet Arslan" <io...@yahoo.com.INVALID> wrote:
>Hi,
>
>I think, xml query parser examples [1] are the safest way to persist
>Lucene queries.
>
>
>[1]https://github.com/apache/lucene-solr/tree/master/lucene/queryparser/sr
>c/test/org/apache/lucene/queryparser/xml
>
>Ahmet
>
>
>On Friday, March 18, 2016 4:02 PM, "Bauer, Herbert S. (Scott)"
><Ba...@mayo.edu> wrote:
>Has anyone in this group solved the problem of serializing complex
>boolean queries (Some of our clauses have span and other query types)?
>Our Java RMI depends upon being able to do so. I have seen posts that
>say you can just parse the string representation but apparently that only
>works on simple query representations. I¹m looking at the CoreParser
>and it¹s supporting xml parsing capabilities with an eye toward
>Marshalling the boolean query into a DOM object and unmarshalling it on
>the server side using some of the support implied by the CoreParser and
>related classes. -scott
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Serializing Queries
Posted by Ahmet Arslan <io...@yahoo.com.INVALID>.
Hi,
I think, xml query parser examples [1] are the safest way to persist Lucene queries.
[1]https://github.com/apache/lucene-solr/tree/master/lucene/queryparser/src/test/org/apache/lucene/queryparser/xml
Ahmet
On Friday, March 18, 2016 4:02 PM, "Bauer, Herbert S. (Scott)" <Ba...@mayo.edu> wrote:
Has anyone in this group solved the problem of serializing complex boolean queries (Some of our clauses have span and other query types)? Our Java RMI depends upon being able to do so. I have seen posts that say you can just parse the string representation but apparently that only works on simple query representations. I’m looking at the CoreParser and it’s supporting xml parsing capabilities with an eye toward Marshalling the boolean query into a DOM object and unmarshalling it on the server side using some of the support implied by the CoreParser and related classes. -scott
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org