You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Jonathan Woods (JIRA)" <ji...@apache.org> on 2007/08/08 12:22:59 UTC
[jira] Created: (SOLR-328) Proposal: ObjectField for storing and
retrieving arbitrary serializable Objects
Proposal: ObjectField for storing and retrieving arbitrary serializable Objects
-------------------------------------------------------------------------------
Key: SOLR-328
URL: https://issues.apache.org/jira/browse/SOLR-328
Project: Solr
Issue Type: New Feature
Components: search
Affects Versions: 1.3
Reporter: Jonathan Woods
Priority: Minor
A while back I developed a means of storing and retrieving arbitrary Objects in a Lucene Document [Field], and I thought that something similar might be useful for Solr. Clearly, it will have more use in embedded Solr implementations, but there's no reason why remote clients couldn't share Object implementations and benefit too.
I wanted to attach a draft implementation of ObjectField, a subclass of org.apache.solr.schema.FieldType, but I soon got lost in a sea of Solr Strings. Instead, I've ended this message with code for getting and setting Object values in Lucene Fields. Someone who's closer to Solr could easily include this in an implementation of ObjectField.
The approach uses Java serialisation, which is often seen as fragile in the sense that changes in class implementation can easily break the serialisation format. However, imo that doesn't matter at all here: if a class's implementation changes all you'd need to do is re-index; and in any case object structure changes are nowhere near as common as object content changes, which are bread and butter to Solr.
Comments welcomed. Oh, and since this is my first communication with Solr - thanks to all concerned for a great piece of software.
Jon
========================================================
public static Object getObject(final Document document, final String fieldName) {
final byte[] serialisation = document.getBinaryValue(fieldName);
try {
return new ObjectInputStream(new ByteArrayInputStream(serialisation)).readObject();
}
catch (final Exception e) {
throw new SearchRuntimeException("While trying to deserialise object from Field", e);
}
}
public static void indexObject(final Document document, final String fieldName, final Serializable object, final boolean compress) throws IndexingException {
final ByteArrayOutputStream boas = new ByteArrayOutputStream();
try {
ObjectOutputStream oos = new ObjectOutputStream(boas);
oos.writeObject(object);
oos.close();
}
catch (IOException e) {
throw new IndexingException("On trying to serialise object", e);
}
document.add(new Field(fieldName, boas.toByteArray(), compress ? Store.COMPRESS : Store.YES));
}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-328) Proposal: ObjectField for storing and
retrieving arbitrary serializable Objects
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12518819 ]
Yonik Seeley commented on SOLR-328:
-----------------------------------
> but I soon got lost in a sea of Solr Strings.
Yes, some of the Strings are unfortunate... this was done before Lucene supported byte[] for stored fields.
Hopefully some day Lucene will support byte[] for indexed values as well, and then we'll really need a bit of a change :-)
But it seems like a generic BinaryFieldType would be more broadly usable., and users could store whatever they want in it. Solr could optionally even support indexing of a binary field even before lucene supports that (if stored-only, use lucene binary field, otherwise transform to string)
FieldType now has toObject, which could efficiently return a byte[] for embedded use.
> Proposal: ObjectField for storing and retrieving arbitrary serializable Objects
> -------------------------------------------------------------------------------
>
> Key: SOLR-328
> URL: https://issues.apache.org/jira/browse/SOLR-328
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.3
> Reporter: Jonathan Woods
> Priority: Minor
>
> A while back I developed a means of storing and retrieving arbitrary Objects in a Lucene Document [Field], and I thought that something similar might be useful for Solr. Clearly, it will have more use in embedded Solr implementations, but there's no reason why remote clients couldn't share Object implementations and benefit too.
> I wanted to attach a draft implementation of ObjectField, a subclass of org.apache.solr.schema.FieldType, but I soon got lost in a sea of Solr Strings. Instead, I've ended this message with code for getting and setting Object values in Lucene Fields. Someone who's closer to Solr could easily include this in an implementation of ObjectField.
> The approach uses Java serialisation, which is often seen as fragile in the sense that changes in class implementation can easily break the serialisation format. However, imo that doesn't matter at all here: if a class's implementation changes all you'd need to do is re-index; and in any case object structure changes are nowhere near as common as object content changes, which are bread and butter to Solr.
> Comments welcomed. Oh, and since this is my first communication with Solr - thanks to all concerned for a great piece of software.
> Jon
> ========================================================
> public static Object getObject(final Document document, final String fieldName) {
> final byte[] serialisation = document.getBinaryValue(fieldName);
> try {
> return new ObjectInputStream(new ByteArrayInputStream(serialisation)).readObject();
> }
> catch (final Exception e) {
> throw new SearchRuntimeException("While trying to deserialise object from Field", e);
> }
> }
> public static void indexObject(final Document document, final String fieldName, final Serializable object, final boolean compress) throws IndexingException {
> final ByteArrayOutputStream boas = new ByteArrayOutputStream();
> try {
> ObjectOutputStream oos = new ObjectOutputStream(boas);
> oos.writeObject(object);
> oos.close();
> }
> catch (IOException e) {
> throw new IndexingException("On trying to serialise object", e);
> }
> document.add(new Field(fieldName, boas.toByteArray(), compress ? Store.COMPRESS : Store.YES));
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-328) Proposal: ObjectField for storing and
retrieving arbitrary serializable Objects
Posted by "Jonathan Woods (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Woods updated SOLR-328:
--------------------------------
Attachment: ObjectField.patch
For what it's worth, I've attached a first attempt at an implementation of ObjectField, as a sub-class of FieldType, for the Solr-orchestrated storage and retrieval of serialisable objects in Lucene fields. I considered naming the class BinaryField, but this really is about [de-]serialisation to and from serialisable Objects, not about generic binary storage.
I'm sure there's plenty wrong with the semantics of the implementation, and I've not yet been able to use it in anger, but it might point up something useful. At any rate, I look forward to being able to do in Solr-world what I'm currently doing using Lucene:
for (Hit hit: searchResults) { Book book = (Book) LuceneDocumentHandler.getObject(hit.getDocument(), STORED_OBJECT_FIELD_NAME); bookDetailsRenderer.render(book); }
I've found rehydrating objects from Lucene a lot faster than regenerating them from my CMS's XML representation. I'm not implying that stashing objects in Documents is always the way to go, just that it makes some difficult things trivially easy.
> Proposal: ObjectField for storing and retrieving arbitrary serializable Objects
> -------------------------------------------------------------------------------
>
> Key: SOLR-328
> URL: https://issues.apache.org/jira/browse/SOLR-328
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.3
> Reporter: Jonathan Woods
> Priority: Minor
> Attachments: ObjectField.patch
>
>
> A while back I developed a means of storing and retrieving arbitrary Objects in a Lucene Document [Field], and I thought that something similar might be useful for Solr. Clearly, it will have more use in embedded Solr implementations, but there's no reason why remote clients couldn't share Object implementations and benefit too.
> I wanted to attach a draft implementation of ObjectField, a subclass of org.apache.solr.schema.FieldType, but I soon got lost in a sea of Solr Strings. Instead, I've ended this message with code for getting and setting Object values in Lucene Fields. Someone who's closer to Solr could easily include this in an implementation of ObjectField.
> The approach uses Java serialisation, which is often seen as fragile in the sense that changes in class implementation can easily break the serialisation format. However, imo that doesn't matter at all here: if a class's implementation changes all you'd need to do is re-index; and in any case object structure changes are nowhere near as common as object content changes, which are bread and butter to Solr.
> Comments welcomed. Oh, and since this is my first communication with Solr - thanks to all concerned for a great piece of software.
> Jon
> ========================================================
> public static Object getObject(final Document document, final String fieldName) {
> final byte[] serialisation = document.getBinaryValue(fieldName);
> try {
> return new ObjectInputStream(new ByteArrayInputStream(serialisation)).readObject();
> }
> catch (final Exception e) {
> throw new SearchRuntimeException("While trying to deserialise object from Field", e);
> }
> }
> public static void indexObject(final Document document, final String fieldName, final Serializable object, final boolean compress) throws IndexingException {
> final ByteArrayOutputStream boas = new ByteArrayOutputStream();
> try {
> ObjectOutputStream oos = new ObjectOutputStream(boas);
> oos.writeObject(object);
> oos.close();
> }
> catch (IOException e) {
> throw new IndexingException("On trying to serialise object", e);
> }
> document.add(new Field(fieldName, boas.toByteArray(), compress ? Store.COMPRESS : Store.YES));
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.