You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Daniel Papasian <Da...@chronicle.com> on 2008/04/17 05:43:24 UTC
XSLT transform before update?
Hey everyone,
I'm experimenting with updating solr from a remote XML source, using an
XSLT transform to get it into the solr XML syntax (and yes, I've looked
into SOLR-469, but disregarded it as I need to do quite a bit using XSLT
to get it to what I can index) to let me maintain an index.
I'm looking at using stream.url, but I need to do the XSLT at some point
in there. I would prefer to do the XSLT on the client (solr) side of
the transfer, for various reasons.
Is there a way to implement a custom request handler or similar to get
solr to apply an XSLT transform to the content stream before it attempts
to parse it? If not possible OOTB, where would be the right place to
add said functionality?
Thanks much for your help,
Daniel
Re: XSLT transform before update?
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
hi ,
There is this new patch which implements these features. I shall
update the wiki with the documentation
I guess we do not need to be too worried about the memory consumption.
A few MB of memory should be fine (unless your are using a file which
is in 10's of MB ). Consider using XPathEntityProcessor (if possible )
it uses Stax and it is pretty efficient.
thanks for your support
--Noble
A few MB of memory for an xml must be fine. The XPathEnt
On Mon, Apr 21, 2008 at 5:57 PM, David Smiley @MITRE.org
<DS...@mitre.org> wrote:
>
> Cool. So you're saying that this xslt file will operate on the entire XML
> document that was fetched from the URL and just pass it on to solr? Thanks
> for supporting this. The XML files I have coming from the my data source
> are big but not not too big to risk an out-of-memory error. And I've found
> xslt to perform fast for me. I like your proposed TemplateTransformer
> too... I'm tempted to use that in place of XSLT. Great job Paul.
>
> It'd be neat to have an XSLT transformer for your framework that operates on
> a single entity (that addresses the memory usage problem). I know your
> entities are HashMap based instead of XML, however.
>
> ~ David
>
>
>
>
> Noble Paul നോബിള് नोब्ळ् wrote:
> >
> > We are planning to incorporate both your requests in the next patch.
> > The implementation is going to be as follows.mention the xsl file
> > location as follows
> > <entity processor="XPathEntitityprocessor xslt="file:/c:/my-own.xsl">
> > ....
> > </entity>
> > So the processing will be done after the XSL transformation. If after
> > your XSL transformation it produces a valid 'add' document not even
> > fields is necessary. Otherwise you will need to write all the fields
> > and their xpaths like any other xml
> >
> > <entity processor="XPathEntitityprocessor xslt="file:/c:/my-own.xsl"
> > useSolrAddXml="true"/>
> >
> > So it will assume that the schema is same as that of the add xml and
> > does the needful.
> >
> > Another feature is going to be a TemplateTransformer which takes in a
> > Template as follows
> >
> > <entity name="e" transformer="TemplateTransformer" ....>
> > <field column="field1_2" template="${e.field1} ${e.field2}/>
> > </entity>
> >
> > Please let us know what u think about this.
> >
> > And keep giving us these great use-cases so that we can make the tool
> > better.
> > --Noble
> >
> >
> >
> > On Mon, Apr 21, 2008 at 12:07 AM, David Smiley @MITRE.org
> > <DS...@mitre.org> wrote:
> >>
> >> Thanks Shalin.
> >>
> >> The particular XSLT processor used is not relevant; it's a spec. Just
> >> use
> >> the standard Java APIs. If I want a particular processor, then I can
> >> get
> >> that to happen by using a system property and/or you could offer a
> >> configuration input for the standard factory class implementation for a
> >> processor of my choice.
> >>
> >> ~ David
> >>
> >>
> >>
> >>
> >> Shalin Shekhar Mangar wrote:
> >> >
> >> > Hi David,
> >> > Actually you can concatenate values, however you'll have to write a
> >> bit of
> >> > code. You can write this in javascript (if you're using Java 6) or in
> >> > Java.
> >> >
> >> > Basically, you need to write a Transformer to do it. Look at
> >> >
> >> http://wiki.apache.org/solr/DataImportHandler#head-a6916b30b5d7605a990fb03c4ff461b3736496a9
> >> >
> >> > For example, lets say you get fields first-name and last-name in the
> >> XML.
> >> > But in the schema.xml you have a field called "name" in which you need
> >> to
> >> > concatenate the values of first-name and last-name (with a space in
> >> > between). Create a Java class:
> >> >
> >> > public class ConcatenateTransformer { public Object
> >> > transformRow(Map<String,
> >> > Object> row) { String firstName = row.get("first-name"); String
> >> lastName =
> >> > row.get("last-name"); row.put("name", firstName + " " + lastName);
> >> return
> >> > row; } }
> >> >
> >> > Add this class to solr's classpath by putting its jar in
> >> solr/WEB-INF/lib
> >> >
> >> > The data-config.xml should like this:
> >> > <entity name="myEntity" processor="XPathEntityProcessor" url="
> >> > http://myurl/example.xml"
> >> > transformer="com.yourpackage.ConcatenateTransformer"> <field
> >> > column="first-name" xpath="/record/first-name" /> <field
> >> > column="last-name"
> >> > xpath="/record/last-name" /> <field column="name" /> </entity>
> >> >
> >> > This will call ConcatenateTransformer.transformRow method for each row
> >> and
> >> > you can concatenate any field with any field (or constant). Note that
> >> solr
> >> > document will keep only those fields which are in the schema.xml, the
> >> rest
> >> > are thrown away.
> >> >
> >> > If you don't want to write this in Java, you can use JavaScript by
> >> using
> >> > the
> >> > built-in ScriptTransformer, for an example look at
> >> >
> >> http://wiki.apache.org/solr/DataImportHandler#head-27fcc2794bd71f7d727104ffc6b99e194bdb6ff9
> >> >
> >> > However, I'm beginning to realize that XSLT is a common need, let me
> >> see
> >> > how
> >> > best we can accomodate it in DataImportHandler. Which XSLT processor
> >> will
> >> > you prefer?
> >> >
> >> > On Sat, Apr 19, 2008 at 12:13 AM, David Smiley @MITRE.org
> >> > <DS...@mitre.org>
> >> > wrote:
> >> >
> >> >>
> >> >> I'm in the same situation as you Daniel. The DataImportHandler is
> >> pretty
> >> >> awesome but I'd also prefer it had the power of XSLT. The XPath
> >> support
> >> >> in
> >> >> it doesn't suffice for me. And I can't do very basic things like
> >> >> concatenate one value with another, say a constant even. It's too
> >> bad
> >> >> there
> >> >> isn't a mode that XSLT can be put in to to not build the whole file
> >> into
> >> >> memory to do the transform. I've been looking into this and have
> >> turned
> >> >> up
> >> >> nothing. It would be neat if there was a STaX to multi-document
> >> adapter,
> >> >> at
> >> >> which point XSLT could be applied to the smaller fixed-size documents
> >> >> instead of the entire data stream. I haven't found anything like
> >> this so
> >> >> it'd need to be built. For now my documents aren't too big to XSLT
> >> >> in-memory.
> >> >>
> >> >> ~ David
> >> >>
> >> >>
> >> >> Daniel Papasian wrote:
> >> >> >
> >> >> > Shalin Shekhar Mangar wrote:
> >> >> >> Hi Daniel,
> >> >> >>
> >> >> >> Maybe if you can give us a sample of how your XML looks like, we
> >> can
> >> >> >> suggest
> >> >> >> how to use SOLR-469 (Data Import Handler) to index it. Most of the
> >> >> >> use-cases
> >> >> >> we have yet encountered are solvable using the
> >> XPathEntityProcessor in
> >> >> >> DataImportHandler without using XSLT, for details look at
> >> >> >>
> >> >>
> >> http://wiki.apache.org/solr/DataImportHandler#head-e68aa93c9ca7b8d261cede2bf1d6110ab1725476
> >> >> >
> >> >> > I think even if it is possible to use SOLR-469 for my needs, I'd
> >> still
> >> >> > prefer the XSLT approach, because it's going to be a bit of
> >> >> > configuration either way, and I'd rather it be an XSLT stylesheet
> >> than
> >> >> > solrconfig.xml. In addition, I haven't yet decided whether I want
> >> to
> >> >> > apply any patches to the version that we will deploy, but if I do
> >> go
> >> >> > down the route of the XSLT transform patch, if I end up having to
> >> back
> >> >> > it out the amount of work that it would be for me to do the
> >> transform
> >> >> at
> >> >> > the XML source would be negligible, where it would be quite a bit
> >> of
> >> >> > work ahead of me to go from using the DataImportHandler to not
> >> using it
> >> >> > at all.
> >> >> >
> >> >> > Because both the solr instance and the XML source are in house, I
> >> have
> >> >> > the ability to apply the XSLT at the source instead of at solr.
> >> >> > However, there are different teams of people that control the XML
> >> >> source
> >> >> > and solr, so it would require a bit more office coordination to do
> >> it
> >> >> on
> >> >> > the backend.
> >> >> >
> >> >> > The data is a filemaker XML export (DTD fmresultset) and it looks
> >> >> > roughly like this:
> >> >> > <fmresultset>
> >> >> > <resultset>
> >> >> > <field name="ID"><data>125</data></field>
> >> >> > <field name="organization"><data>Ford
> >> Foundation</data></field>
> >> >> > ...
> >> >> > <relatedset table="Employees">
> >> >> > <record>
> >> >> > <field name="ID"><data>Y5-A</data></field>
> >> >> > <field name="Name"><data>John Smith</data></field>
> >> >> > </record>
> >> >> > <record>
> >> >> > <field name="ID"><data>Y5-B</data></field>
> >> >> > <field name="Name"><data>Jane Doe</data></field>
> >> >> > </record>
> >> >> > </relatedset>
> >> >> > </fmresultset>
> >> >> >
> >> >> > I'm taking the product of the resultset and the relatedset, using
> >> both
> >> >> > IDs concatenated as a unique identifier, like so:
> >> >> >
> >> >> > <doc>
> >> >> > <field name="ID">125Y5-A</field>
> >> >> > <field name="organization">Ford Foundation</field>
> >> >> > <field name="Name">John Smith</field>
> >> >> > </doc>
> >> >> > <doc>
> >> >> > <field name="ID">125Y5-B</field>
> >> >> > <field name="organization">Ford Foundation</field>
> >> >> > <field name="Name">Jane Doe</field>
> >> >> > </doc>
> >> >> >
> >> >> > I can do the transform pretty simply with XSLT. I suppose it is
> >> >> > possible to get the DataImportHandler to do this, but I'm not yet
> >> >> > convinced that it's easier.
> >> >> >
> >> >> > Daniel
> >> >> >
> >> >> >
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >> http://www.nabble.com/XSLT-transform-before-update--tp16738227p16764009.html
> >> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Shalin Shekhar Mangar.
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >> http://www.nabble.com/XSLT-transform-before-update--tp16738227p16796900.html
> >>
> >>
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context: http://www.nabble.com/XSLT-transform-before-update--tp16738227p16807488.html
>
>
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: XSLT transform before update?
Posted by "David Smiley @MITRE.org" <DS...@mitre.org>.
Cool. So you're saying that this xslt file will operate on the entire XML
document that was fetched from the URL and just pass it on to solr? Thanks
for supporting this. The XML files I have coming from the my data source
are big but not not too big to risk an out-of-memory error. And I've found
xslt to perform fast for me. I like your proposed TemplateTransformer
too... I'm tempted to use that in place of XSLT. Great job Paul.
It'd be neat to have an XSLT transformer for your framework that operates on
a single entity (that addresses the memory usage problem). I know your
entities are HashMap based instead of XML, however.
~ David
Noble Paul നോബിള് नोब्ळ् wrote:
>
> We are planning to incorporate both your requests in the next patch.
> The implementation is going to be as follows.mention the xsl file
> location as follows
> <entity processor="XPathEntitityprocessor xslt="file:/c:/my-own.xsl">
> ....
> </entity>
> So the processing will be done after the XSL transformation. If after
> your XSL transformation it produces a valid 'add' document not even
> fields is necessary. Otherwise you will need to write all the fields
> and their xpaths like any other xml
>
> <entity processor="XPathEntitityprocessor xslt="file:/c:/my-own.xsl"
> useSolrAddXml="true"/>
>
> So it will assume that the schema is same as that of the add xml and
> does the needful.
>
> Another feature is going to be a TemplateTransformer which takes in a
> Template as follows
>
> <entity name="e" transformer="TemplateTransformer" ....>
> <field column="field1_2" template="${e.field1} ${e.field2}/>
> </entity>
>
> Please let us know what u think about this.
>
> And keep giving us these great use-cases so that we can make the tool
> better.
> --Noble
>
>
>
> On Mon, Apr 21, 2008 at 12:07 AM, David Smiley @MITRE.org
> <DS...@mitre.org> wrote:
>>
>> Thanks Shalin.
>>
>> The particular XSLT processor used is not relevant; it's a spec. Just
>> use
>> the standard Java APIs. If I want a particular processor, then I can
>> get
>> that to happen by using a system property and/or you could offer a
>> configuration input for the standard factory class implementation for a
>> processor of my choice.
>>
>> ~ David
>>
>>
>>
>>
>> Shalin Shekhar Mangar wrote:
>> >
>> > Hi David,
>> > Actually you can concatenate values, however you'll have to write a
>> bit of
>> > code. You can write this in javascript (if you're using Java 6) or in
>> > Java.
>> >
>> > Basically, you need to write a Transformer to do it. Look at
>> >
>> http://wiki.apache.org/solr/DataImportHandler#head-a6916b30b5d7605a990fb03c4ff461b3736496a9
>> >
>> > For example, lets say you get fields first-name and last-name in the
>> XML.
>> > But in the schema.xml you have a field called "name" in which you need
>> to
>> > concatenate the values of first-name and last-name (with a space in
>> > between). Create a Java class:
>> >
>> > public class ConcatenateTransformer { public Object
>> > transformRow(Map<String,
>> > Object> row) { String firstName = row.get("first-name"); String
>> lastName =
>> > row.get("last-name"); row.put("name", firstName + " " + lastName);
>> return
>> > row; } }
>> >
>> > Add this class to solr's classpath by putting its jar in
>> solr/WEB-INF/lib
>> >
>> > The data-config.xml should like this:
>> > <entity name="myEntity" processor="XPathEntityProcessor" url="
>> > http://myurl/example.xml"
>> > transformer="com.yourpackage.ConcatenateTransformer"> <field
>> > column="first-name" xpath="/record/first-name" /> <field
>> > column="last-name"
>> > xpath="/record/last-name" /> <field column="name" /> </entity>
>> >
>> > This will call ConcatenateTransformer.transformRow method for each row
>> and
>> > you can concatenate any field with any field (or constant). Note that
>> solr
>> > document will keep only those fields which are in the schema.xml, the
>> rest
>> > are thrown away.
>> >
>> > If you don't want to write this in Java, you can use JavaScript by
>> using
>> > the
>> > built-in ScriptTransformer, for an example look at
>> >
>> http://wiki.apache.org/solr/DataImportHandler#head-27fcc2794bd71f7d727104ffc6b99e194bdb6ff9
>> >
>> > However, I'm beginning to realize that XSLT is a common need, let me
>> see
>> > how
>> > best we can accomodate it in DataImportHandler. Which XSLT processor
>> will
>> > you prefer?
>> >
>> > On Sat, Apr 19, 2008 at 12:13 AM, David Smiley @MITRE.org
>> > <DS...@mitre.org>
>> > wrote:
>> >
>> >>
>> >> I'm in the same situation as you Daniel. The DataImportHandler is
>> pretty
>> >> awesome but I'd also prefer it had the power of XSLT. The XPath
>> support
>> >> in
>> >> it doesn't suffice for me. And I can't do very basic things like
>> >> concatenate one value with another, say a constant even. It's too
>> bad
>> >> there
>> >> isn't a mode that XSLT can be put in to to not build the whole file
>> into
>> >> memory to do the transform. I've been looking into this and have
>> turned
>> >> up
>> >> nothing. It would be neat if there was a STaX to multi-document
>> adapter,
>> >> at
>> >> which point XSLT could be applied to the smaller fixed-size documents
>> >> instead of the entire data stream. I haven't found anything like
>> this so
>> >> it'd need to be built. For now my documents aren't too big to XSLT
>> >> in-memory.
>> >>
>> >> ~ David
>> >>
>> >>
>> >> Daniel Papasian wrote:
>> >> >
>> >> > Shalin Shekhar Mangar wrote:
>> >> >> Hi Daniel,
>> >> >>
>> >> >> Maybe if you can give us a sample of how your XML looks like, we
>> can
>> >> >> suggest
>> >> >> how to use SOLR-469 (Data Import Handler) to index it. Most of the
>> >> >> use-cases
>> >> >> we have yet encountered are solvable using the
>> XPathEntityProcessor in
>> >> >> DataImportHandler without using XSLT, for details look at
>> >> >>
>> >>
>> http://wiki.apache.org/solr/DataImportHandler#head-e68aa93c9ca7b8d261cede2bf1d6110ab1725476
>> >> >
>> >> > I think even if it is possible to use SOLR-469 for my needs, I'd
>> still
>> >> > prefer the XSLT approach, because it's going to be a bit of
>> >> > configuration either way, and I'd rather it be an XSLT stylesheet
>> than
>> >> > solrconfig.xml. In addition, I haven't yet decided whether I want
>> to
>> >> > apply any patches to the version that we will deploy, but if I do
>> go
>> >> > down the route of the XSLT transform patch, if I end up having to
>> back
>> >> > it out the amount of work that it would be for me to do the
>> transform
>> >> at
>> >> > the XML source would be negligible, where it would be quite a bit
>> of
>> >> > work ahead of me to go from using the DataImportHandler to not
>> using it
>> >> > at all.
>> >> >
>> >> > Because both the solr instance and the XML source are in house, I
>> have
>> >> > the ability to apply the XSLT at the source instead of at solr.
>> >> > However, there are different teams of people that control the XML
>> >> source
>> >> > and solr, so it would require a bit more office coordination to do
>> it
>> >> on
>> >> > the backend.
>> >> >
>> >> > The data is a filemaker XML export (DTD fmresultset) and it looks
>> >> > roughly like this:
>> >> > <fmresultset>
>> >> > <resultset>
>> >> > <field name="ID"><data>125</data></field>
>> >> > <field name="organization"><data>Ford
>> Foundation</data></field>
>> >> > ...
>> >> > <relatedset table="Employees">
>> >> > <record>
>> >> > <field name="ID"><data>Y5-A</data></field>
>> >> > <field name="Name"><data>John Smith</data></field>
>> >> > </record>
>> >> > <record>
>> >> > <field name="ID"><data>Y5-B</data></field>
>> >> > <field name="Name"><data>Jane Doe</data></field>
>> >> > </record>
>> >> > </relatedset>
>> >> > </fmresultset>
>> >> >
>> >> > I'm taking the product of the resultset and the relatedset, using
>> both
>> >> > IDs concatenated as a unique identifier, like so:
>> >> >
>> >> > <doc>
>> >> > <field name="ID">125Y5-A</field>
>> >> > <field name="organization">Ford Foundation</field>
>> >> > <field name="Name">John Smith</field>
>> >> > </doc>
>> >> > <doc>
>> >> > <field name="ID">125Y5-B</field>
>> >> > <field name="organization">Ford Foundation</field>
>> >> > <field name="Name">Jane Doe</field>
>> >> > </doc>
>> >> >
>> >> > I can do the transform pretty simply with XSLT. I suppose it is
>> >> > possible to get the DataImportHandler to do this, but I'm not yet
>> >> > convinced that it's easier.
>> >> >
>> >> > Daniel
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/XSLT-transform-before-update--tp16738227p16764009.html
>> >> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> > Regards,
>> > Shalin Shekhar Mangar.
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/XSLT-transform-before-update--tp16738227p16796900.html
>>
>>
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/XSLT-transform-before-update--tp16738227p16807488.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: XSLT transform before update?
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
We are planning to incorporate both your requests in the next patch.
The implementation is going to be as follows.mention the xsl file
location as follows
<entity processor="XPathEntitityprocessor xslt="file:/c:/my-own.xsl">
....
</entity>
So the processing will be done after the XSL transformation. If after
your XSL transformation it produces a valid 'add' document not even
fields is necessary. Otherwise you will need to write all the fields
and their xpaths like any other xml
<entity processor="XPathEntitityprocessor xslt="file:/c:/my-own.xsl"
useSolrAddXml="true"/>
So it will assume that the schema is same as that of the add xml and
does the needful.
Another feature is going to be a TemplateTransformer which takes in a
Template as follows
<entity name="e" transformer="TemplateTransformer" ....>
<field column="field1_2" template="${e.field1} ${e.field2}/>
</entity>
Please let us know what u think about this.
And keep giving us these great use-cases so that we can make the tool better.
--Noble
On Mon, Apr 21, 2008 at 12:07 AM, David Smiley @MITRE.org
<DS...@mitre.org> wrote:
>
> Thanks Shalin.
>
> The particular XSLT processor used is not relevant; it's a spec. Just use
> the standard Java APIs. If I want a particular processor, then I can get
> that to happen by using a system property and/or you could offer a
> configuration input for the standard factory class implementation for a
> processor of my choice.
>
> ~ David
>
>
>
>
> Shalin Shekhar Mangar wrote:
> >
> > Hi David,
> > Actually you can concatenate values, however you'll have to write a bit of
> > code. You can write this in javascript (if you're using Java 6) or in
> > Java.
> >
> > Basically, you need to write a Transformer to do it. Look at
> > http://wiki.apache.org/solr/DataImportHandler#head-a6916b30b5d7605a990fb03c4ff461b3736496a9
> >
> > For example, lets say you get fields first-name and last-name in the XML.
> > But in the schema.xml you have a field called "name" in which you need to
> > concatenate the values of first-name and last-name (with a space in
> > between). Create a Java class:
> >
> > public class ConcatenateTransformer { public Object
> > transformRow(Map<String,
> > Object> row) { String firstName = row.get("first-name"); String lastName =
> > row.get("last-name"); row.put("name", firstName + " " + lastName); return
> > row; } }
> >
> > Add this class to solr's classpath by putting its jar in solr/WEB-INF/lib
> >
> > The data-config.xml should like this:
> > <entity name="myEntity" processor="XPathEntityProcessor" url="
> > http://myurl/example.xml"
> > transformer="com.yourpackage.ConcatenateTransformer"> <field
> > column="first-name" xpath="/record/first-name" /> <field
> > column="last-name"
> > xpath="/record/last-name" /> <field column="name" /> </entity>
> >
> > This will call ConcatenateTransformer.transformRow method for each row and
> > you can concatenate any field with any field (or constant). Note that solr
> > document will keep only those fields which are in the schema.xml, the rest
> > are thrown away.
> >
> > If you don't want to write this in Java, you can use JavaScript by using
> > the
> > built-in ScriptTransformer, for an example look at
> > http://wiki.apache.org/solr/DataImportHandler#head-27fcc2794bd71f7d727104ffc6b99e194bdb6ff9
> >
> > However, I'm beginning to realize that XSLT is a common need, let me see
> > how
> > best we can accomodate it in DataImportHandler. Which XSLT processor will
> > you prefer?
> >
> > On Sat, Apr 19, 2008 at 12:13 AM, David Smiley @MITRE.org
> > <DS...@mitre.org>
> > wrote:
> >
> >>
> >> I'm in the same situation as you Daniel. The DataImportHandler is pretty
> >> awesome but I'd also prefer it had the power of XSLT. The XPath support
> >> in
> >> it doesn't suffice for me. And I can't do very basic things like
> >> concatenate one value with another, say a constant even. It's too bad
> >> there
> >> isn't a mode that XSLT can be put in to to not build the whole file into
> >> memory to do the transform. I've been looking into this and have turned
> >> up
> >> nothing. It would be neat if there was a STaX to multi-document adapter,
> >> at
> >> which point XSLT could be applied to the smaller fixed-size documents
> >> instead of the entire data stream. I haven't found anything like this so
> >> it'd need to be built. For now my documents aren't too big to XSLT
> >> in-memory.
> >>
> >> ~ David
> >>
> >>
> >> Daniel Papasian wrote:
> >> >
> >> > Shalin Shekhar Mangar wrote:
> >> >> Hi Daniel,
> >> >>
> >> >> Maybe if you can give us a sample of how your XML looks like, we can
> >> >> suggest
> >> >> how to use SOLR-469 (Data Import Handler) to index it. Most of the
> >> >> use-cases
> >> >> we have yet encountered are solvable using the XPathEntityProcessor in
> >> >> DataImportHandler without using XSLT, for details look at
> >> >>
> >> http://wiki.apache.org/solr/DataImportHandler#head-e68aa93c9ca7b8d261cede2bf1d6110ab1725476
> >> >
> >> > I think even if it is possible to use SOLR-469 for my needs, I'd still
> >> > prefer the XSLT approach, because it's going to be a bit of
> >> > configuration either way, and I'd rather it be an XSLT stylesheet than
> >> > solrconfig.xml. In addition, I haven't yet decided whether I want to
> >> > apply any patches to the version that we will deploy, but if I do go
> >> > down the route of the XSLT transform patch, if I end up having to back
> >> > it out the amount of work that it would be for me to do the transform
> >> at
> >> > the XML source would be negligible, where it would be quite a bit of
> >> > work ahead of me to go from using the DataImportHandler to not using it
> >> > at all.
> >> >
> >> > Because both the solr instance and the XML source are in house, I have
> >> > the ability to apply the XSLT at the source instead of at solr.
> >> > However, there are different teams of people that control the XML
> >> source
> >> > and solr, so it would require a bit more office coordination to do it
> >> on
> >> > the backend.
> >> >
> >> > The data is a filemaker XML export (DTD fmresultset) and it looks
> >> > roughly like this:
> >> > <fmresultset>
> >> > <resultset>
> >> > <field name="ID"><data>125</data></field>
> >> > <field name="organization"><data>Ford Foundation</data></field>
> >> > ...
> >> > <relatedset table="Employees">
> >> > <record>
> >> > <field name="ID"><data>Y5-A</data></field>
> >> > <field name="Name"><data>John Smith</data></field>
> >> > </record>
> >> > <record>
> >> > <field name="ID"><data>Y5-B</data></field>
> >> > <field name="Name"><data>Jane Doe</data></field>
> >> > </record>
> >> > </relatedset>
> >> > </fmresultset>
> >> >
> >> > I'm taking the product of the resultset and the relatedset, using both
> >> > IDs concatenated as a unique identifier, like so:
> >> >
> >> > <doc>
> >> > <field name="ID">125Y5-A</field>
> >> > <field name="organization">Ford Foundation</field>
> >> > <field name="Name">John Smith</field>
> >> > </doc>
> >> > <doc>
> >> > <field name="ID">125Y5-B</field>
> >> > <field name="organization">Ford Foundation</field>
> >> > <field name="Name">Jane Doe</field>
> >> > </doc>
> >> >
> >> > I can do the transform pretty simply with XSLT. I suppose it is
> >> > possible to get the DataImportHandler to do this, but I'm not yet
> >> > convinced that it's easier.
> >> >
> >> > Daniel
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >> http://www.nabble.com/XSLT-transform-before-update--tp16738227p16764009.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
> >
> >
>
> --
> View this message in context: http://www.nabble.com/XSLT-transform-before-update--tp16738227p16796900.html
>
>
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: XSLT transform before update?
Posted by "David Smiley @MITRE.org" <DS...@mitre.org>.
Thanks Shalin.
The particular XSLT processor used is not relevant; it's a spec. Just use
the standard Java APIs. If I want a particular processor, then I can get
that to happen by using a system property and/or you could offer a
configuration input for the standard factory class implementation for a
processor of my choice.
~ David
Shalin Shekhar Mangar wrote:
>
> Hi David,
> Actually you can concatenate values, however you'll have to write a bit of
> code. You can write this in javascript (if you're using Java 6) or in
> Java.
>
> Basically, you need to write a Transformer to do it. Look at
> http://wiki.apache.org/solr/DataImportHandler#head-a6916b30b5d7605a990fb03c4ff461b3736496a9
>
> For example, lets say you get fields first-name and last-name in the XML.
> But in the schema.xml you have a field called "name" in which you need to
> concatenate the values of first-name and last-name (with a space in
> between). Create a Java class:
>
> public class ConcatenateTransformer { public Object
> transformRow(Map<String,
> Object> row) { String firstName = row.get("first-name"); String lastName =
> row.get("last-name"); row.put("name", firstName + " " + lastName); return
> row; } }
>
> Add this class to solr's classpath by putting its jar in solr/WEB-INF/lib
>
> The data-config.xml should like this:
> <entity name="myEntity" processor="XPathEntityProcessor" url="
> http://myurl/example.xml"
> transformer="com.yourpackage.ConcatenateTransformer"> <field
> column="first-name" xpath="/record/first-name" /> <field
> column="last-name"
> xpath="/record/last-name" /> <field column="name" /> </entity>
>
> This will call ConcatenateTransformer.transformRow method for each row and
> you can concatenate any field with any field (or constant). Note that solr
> document will keep only those fields which are in the schema.xml, the rest
> are thrown away.
>
> If you don't want to write this in Java, you can use JavaScript by using
> the
> built-in ScriptTransformer, for an example look at
> http://wiki.apache.org/solr/DataImportHandler#head-27fcc2794bd71f7d727104ffc6b99e194bdb6ff9
>
> However, I'm beginning to realize that XSLT is a common need, let me see
> how
> best we can accomodate it in DataImportHandler. Which XSLT processor will
> you prefer?
>
> On Sat, Apr 19, 2008 at 12:13 AM, David Smiley @MITRE.org
> <DS...@mitre.org>
> wrote:
>
>>
>> I'm in the same situation as you Daniel. The DataImportHandler is pretty
>> awesome but I'd also prefer it had the power of XSLT. The XPath support
>> in
>> it doesn't suffice for me. And I can't do very basic things like
>> concatenate one value with another, say a constant even. It's too bad
>> there
>> isn't a mode that XSLT can be put in to to not build the whole file into
>> memory to do the transform. I've been looking into this and have turned
>> up
>> nothing. It would be neat if there was a STaX to multi-document adapter,
>> at
>> which point XSLT could be applied to the smaller fixed-size documents
>> instead of the entire data stream. I haven't found anything like this so
>> it'd need to be built. For now my documents aren't too big to XSLT
>> in-memory.
>>
>> ~ David
>>
>>
>> Daniel Papasian wrote:
>> >
>> > Shalin Shekhar Mangar wrote:
>> >> Hi Daniel,
>> >>
>> >> Maybe if you can give us a sample of how your XML looks like, we can
>> >> suggest
>> >> how to use SOLR-469 (Data Import Handler) to index it. Most of the
>> >> use-cases
>> >> we have yet encountered are solvable using the XPathEntityProcessor in
>> >> DataImportHandler without using XSLT, for details look at
>> >>
>> http://wiki.apache.org/solr/DataImportHandler#head-e68aa93c9ca7b8d261cede2bf1d6110ab1725476
>> >
>> > I think even if it is possible to use SOLR-469 for my needs, I'd still
>> > prefer the XSLT approach, because it's going to be a bit of
>> > configuration either way, and I'd rather it be an XSLT stylesheet than
>> > solrconfig.xml. In addition, I haven't yet decided whether I want to
>> > apply any patches to the version that we will deploy, but if I do go
>> > down the route of the XSLT transform patch, if I end up having to back
>> > it out the amount of work that it would be for me to do the transform
>> at
>> > the XML source would be negligible, where it would be quite a bit of
>> > work ahead of me to go from using the DataImportHandler to not using it
>> > at all.
>> >
>> > Because both the solr instance and the XML source are in house, I have
>> > the ability to apply the XSLT at the source instead of at solr.
>> > However, there are different teams of people that control the XML
>> source
>> > and solr, so it would require a bit more office coordination to do it
>> on
>> > the backend.
>> >
>> > The data is a filemaker XML export (DTD fmresultset) and it looks
>> > roughly like this:
>> > <fmresultset>
>> > <resultset>
>> > <field name="ID"><data>125</data></field>
>> > <field name="organization"><data>Ford Foundation</data></field>
>> > ...
>> > <relatedset table="Employees">
>> > <record>
>> > <field name="ID"><data>Y5-A</data></field>
>> > <field name="Name"><data>John Smith</data></field>
>> > </record>
>> > <record>
>> > <field name="ID"><data>Y5-B</data></field>
>> > <field name="Name"><data>Jane Doe</data></field>
>> > </record>
>> > </relatedset>
>> > </fmresultset>
>> >
>> > I'm taking the product of the resultset and the relatedset, using both
>> > IDs concatenated as a unique identifier, like so:
>> >
>> > <doc>
>> > <field name="ID">125Y5-A</field>
>> > <field name="organization">Ford Foundation</field>
>> > <field name="Name">John Smith</field>
>> > </doc>
>> > <doc>
>> > <field name="ID">125Y5-B</field>
>> > <field name="organization">Ford Foundation</field>
>> > <field name="Name">Jane Doe</field>
>> > </doc>
>> >
>> > I can do the transform pretty simply with XSLT. I suppose it is
>> > possible to get the DataImportHandler to do this, but I'm not yet
>> > convinced that it's easier.
>> >
>> > Daniel
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/XSLT-transform-before-update--tp16738227p16764009.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/XSLT-transform-before-update--tp16738227p16796900.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: better stemming engine than Porter?
Posted by "Wagner,Harry" <wa...@oclc.org>.
Thanks Ryan. I just opened SOLR-546. Please let me know if I can provide
further help. Cheers! h
-----Original Message-----
From: Ryan McKinley [mailto:ryantxu@gmail.com]
Sent: Monday, April 21, 2008 2:33 PM
To: solr-user@lucene.apache.org
Subject: Re: better stemming engine than Porter?
Hey-
to create an issue, make an account on jira and post it...
https://issues.apache.org/jira/browse/SOLR
Give that a try and holler if you have trouble.
ryan
On Apr 21, 2008, at 12:31 PM, Wagner,Harry wrote:
> Hi HH,
> Here's a note I sent Solr-dev a while back:
>
> ---
> I've implemented a Solr plug-in that wraps KStem for Solr use (someone
> else had already written a Lucene wrapper for it). KStem is
> considered
> to be more appropriate for library usage since it is much less
> aggressive than Porter (i.e., searches for organization do NOT match
> on
> organ!). If there is any interest in feeding this back into Solr I
> would
> be happy to contribute it.
> ---
>
> I believe there was interest in it, but I never opened an issue for it
> and I don't know if it was ever followed-up on. I'd be happy to do
> that
> now. Can someone on the Solr-dev team point me in the right direction
> for opening an issue?
>
> Thanks... harry
>
>
> -----Original Message-----
> From: Hung Huynh [mailto:hh@vinaconsulting.com]
> Sent: Monday, April 21, 2008 11:59 AM
> To: solr-user@lucene.apache.org
> Subject: better stemming engine than Porter?
>
> I recall I've read some where in one of the mailing-list archives that
> some
> one had developed a better stemming algo for Solr than the built-in
> Porter
> stemming. Does anyone have link to that stemming module?
>
> Thanks,
>
> HH
>
>
>
Re: better stemming engine than Porter?
Posted by Chris Hostetter <ho...@fucit.org>.
: to create an issue, make an account on jira and post it...
: https://issues.apache.org/jira/browse/SOLR
:
: Give that a try and holler if you have trouble.
To elaborate more (and save some time on the question answering of the
correct procedures) ...
http://wiki.apache.org/solr/HowToContribute
(note the "contributing code" section)
-Hoss
Re: better stemming engine than Porter?
Posted by Ryan McKinley <ry...@gmail.com>.
Hey-
to create an issue, make an account on jira and post it...
https://issues.apache.org/jira/browse/SOLR
Give that a try and holler if you have trouble.
ryan
On Apr 21, 2008, at 12:31 PM, Wagner,Harry wrote:
> Hi HH,
> Here's a note I sent Solr-dev a while back:
>
> ---
> I've implemented a Solr plug-in that wraps KStem for Solr use (someone
> else had already written a Lucene wrapper for it). KStem is
> considered
> to be more appropriate for library usage since it is much less
> aggressive than Porter (i.e., searches for organization do NOT match
> on
> organ!). If there is any interest in feeding this back into Solr I
> would
> be happy to contribute it.
> ---
>
> I believe there was interest in it, but I never opened an issue for it
> and I don't know if it was ever followed-up on. I'd be happy to do
> that
> now. Can someone on the Solr-dev team point me in the right direction
> for opening an issue?
>
> Thanks... harry
>
>
> -----Original Message-----
> From: Hung Huynh [mailto:hh@vinaconsulting.com]
> Sent: Monday, April 21, 2008 11:59 AM
> To: solr-user@lucene.apache.org
> Subject: better stemming engine than Porter?
>
> I recall I've read some where in one of the mailing-list archives that
> some
> one had developed a better stemming algo for Solr than the built-in
> Porter
> stemming. Does anyone have link to that stemming module?
>
> Thanks,
>
> HH
>
>
>
RE: better stemming engine than Porter?
Posted by "Wagner,Harry" <wa...@oclc.org>.
Hi Jay,
I did not do a timing comparison either, but any change in performance after switching to Kstem was not noticeable. Cheers... h
-----Original Message-----
From: Jay [mailto:yu@AI.SRI.COM]
Sent: Tuesday, April 22, 2008 12:26 PM
To: solr-user@lucene.apache.org
Subject: Re: better stemming engine than Porter?
Hi Wagner,
Thanks for the intro of KStem! I quickly scanned the original paper on
KStem by Robert Krovetz but could not find any timing comparison data on
KStem and Porter stem. I wonder how slow/fast Kstem is compared to
Porter stem based on your use in your application?
Jay
Wagner,Harry wrote:
> Mathieu,
> It's not my Kstem. It was written by someone at Umass, Amherst. More info here:
> http://ciir.cs.umass.edu/cgi-bin/downloads/downloads.cgi
>
> Someone else had already ported it to Lucene. I simply modified that wrapper to work with Solr. I'll open an issue for it so that it can (hopefully) be integrated into the project.
>
> Cheers... harry
>
> -----Original Message-----
> From: Mathieu Lecarme [mailto:mathieu@garambrogne.net]
> Sent: Tuesday, April 22, 2008 3:57 AM
> To: solr-user@lucene.apache.org
> Subject: Re: better stemming engine than Porter?
>
> Porter stemmer is not only agressive, it is ugly, too. The generated
> code is too old, too few object centric and should be too slow.
> If your kstem compile with java 1.4, why don't you suggest it to lucene
> core?
>
> M.
>
> Wagner,Harry a écrit :
>> Hi HH,
>> Here's a note I sent Solr-dev a while back:
>>
>> ---
>> I've implemented a Solr plug-in that wraps KStem for Solr use (someone
>> else had already written a Lucene wrapper for it). KStem is considered
>> to be more appropriate for library usage since it is much less
>> aggressive than Porter (i.e., searches for organization do NOT match on
>> organ!). If there is any interest in feeding this back into Solr I would
>> be happy to contribute it.
>> ---
>>
>> I believe there was interest in it, but I never opened an issue for it
>> and I don't know if it was ever followed-up on. I'd be happy to do that
>> now. Can someone on the Solr-dev team point me in the right direction
>> for opening an issue?
>>
>> Thanks... harry
>>
>>
>> -----Original Message-----
>> From: Hung Huynh [mailto:hh@vinaconsulting.com]
>> Sent: Monday, April 21, 2008 11:59 AM
>> To: solr-user@lucene.apache.org
>> Subject: better stemming engine than Porter?
>>
>> I recall I've read some where in one of the mailing-list archives that
>> some
>> one had developed a better stemming algo for Solr than the built-in
>> Porter
>> stemming. Does anyone have link to that stemming module?
>>
>> Thanks,
>>
>> HH
>>
>>
>>
>>
>>
>
>
>
Re: better stemming engine than Porter?
Posted by Jay <yu...@AI.SRI.COM>.
Hi Wagner,
Thanks for the intro of KStem! I quickly scanned the original paper on
KStem by Robert Krovetz but could not find any timing comparison data on
KStem and Porter stem. I wonder how slow/fast Kstem is compared to
Porter stem based on your use in your application?
Jay
Wagner,Harry wrote:
> Mathieu,
> It's not my Kstem. It was written by someone at Umass, Amherst. More info here:
> http://ciir.cs.umass.edu/cgi-bin/downloads/downloads.cgi
>
> Someone else had already ported it to Lucene. I simply modified that wrapper to work with Solr. I'll open an issue for it so that it can (hopefully) be integrated into the project.
>
> Cheers... harry
>
> -----Original Message-----
> From: Mathieu Lecarme [mailto:mathieu@garambrogne.net]
> Sent: Tuesday, April 22, 2008 3:57 AM
> To: solr-user@lucene.apache.org
> Subject: Re: better stemming engine than Porter?
>
> Porter stemmer is not only agressive, it is ugly, too. The generated
> code is too old, too few object centric and should be too slow.
> If your kstem compile with java 1.4, why don't you suggest it to lucene
> core?
>
> M.
>
> Wagner,Harry a écrit :
>> Hi HH,
>> Here's a note I sent Solr-dev a while back:
>>
>> ---
>> I've implemented a Solr plug-in that wraps KStem for Solr use (someone
>> else had already written a Lucene wrapper for it). KStem is considered
>> to be more appropriate for library usage since it is much less
>> aggressive than Porter (i.e., searches for organization do NOT match on
>> organ!). If there is any interest in feeding this back into Solr I would
>> be happy to contribute it.
>> ---
>>
>> I believe there was interest in it, but I never opened an issue for it
>> and I don't know if it was ever followed-up on. I'd be happy to do that
>> now. Can someone on the Solr-dev team point me in the right direction
>> for opening an issue?
>>
>> Thanks... harry
>>
>>
>> -----Original Message-----
>> From: Hung Huynh [mailto:hh@vinaconsulting.com]
>> Sent: Monday, April 21, 2008 11:59 AM
>> To: solr-user@lucene.apache.org
>> Subject: better stemming engine than Porter?
>>
>> I recall I've read some where in one of the mailing-list archives that
>> some
>> one had developed a better stemming algo for Solr than the built-in
>> Porter
>> stemming. Does anyone have link to that stemming module?
>>
>> Thanks,
>>
>> HH
>>
>>
>>
>>
>>
>
>
>
RE: better stemming engine than Porter?
Posted by "Wagner,Harry" <wa...@oclc.org>.
Mathieu,
It's not my Kstem. It was written by someone at Umass, Amherst. More info here:
http://ciir.cs.umass.edu/cgi-bin/downloads/downloads.cgi
Someone else had already ported it to Lucene. I simply modified that wrapper to work with Solr. I'll open an issue for it so that it can (hopefully) be integrated into the project.
Cheers... harry
-----Original Message-----
From: Mathieu Lecarme [mailto:mathieu@garambrogne.net]
Sent: Tuesday, April 22, 2008 3:57 AM
To: solr-user@lucene.apache.org
Subject: Re: better stemming engine than Porter?
Porter stemmer is not only agressive, it is ugly, too. The generated
code is too old, too few object centric and should be too slow.
If your kstem compile with java 1.4, why don't you suggest it to lucene
core?
M.
Wagner,Harry a écrit :
> Hi HH,
> Here's a note I sent Solr-dev a while back:
>
> ---
> I've implemented a Solr plug-in that wraps KStem for Solr use (someone
> else had already written a Lucene wrapper for it). KStem is considered
> to be more appropriate for library usage since it is much less
> aggressive than Porter (i.e., searches for organization do NOT match on
> organ!). If there is any interest in feeding this back into Solr I would
> be happy to contribute it.
> ---
>
> I believe there was interest in it, but I never opened an issue for it
> and I don't know if it was ever followed-up on. I'd be happy to do that
> now. Can someone on the Solr-dev team point me in the right direction
> for opening an issue?
>
> Thanks... harry
>
>
> -----Original Message-----
> From: Hung Huynh [mailto:hh@vinaconsulting.com]
> Sent: Monday, April 21, 2008 11:59 AM
> To: solr-user@lucene.apache.org
> Subject: better stemming engine than Porter?
>
> I recall I've read some where in one of the mailing-list archives that
> some
> one had developed a better stemming algo for Solr than the built-in
> Porter
> stemming. Does anyone have link to that stemming module?
>
> Thanks,
>
> HH
>
>
>
>
>
Re: better stemming engine than Porter?
Posted by Mathieu Lecarme <ma...@garambrogne.net>.
Porter stemmer is not only agressive, it is ugly, too. The generated
code is too old, too few object centric and should be too slow.
If your kstem compile with java 1.4, why don't you suggest it to lucene
core?
M.
Wagner,Harry a écrit :
> Hi HH,
> Here's a note I sent Solr-dev a while back:
>
> ---
> I've implemented a Solr plug-in that wraps KStem for Solr use (someone
> else had already written a Lucene wrapper for it). KStem is considered
> to be more appropriate for library usage since it is much less
> aggressive than Porter (i.e., searches for organization do NOT match on
> organ!). If there is any interest in feeding this back into Solr I would
> be happy to contribute it.
> ---
>
> I believe there was interest in it, but I never opened an issue for it
> and I don't know if it was ever followed-up on. I'd be happy to do that
> now. Can someone on the Solr-dev team point me in the right direction
> for opening an issue?
>
> Thanks... harry
>
>
> -----Original Message-----
> From: Hung Huynh [mailto:hh@vinaconsulting.com]
> Sent: Monday, April 21, 2008 11:59 AM
> To: solr-user@lucene.apache.org
> Subject: better stemming engine than Porter?
>
> I recall I've read some where in one of the mailing-list archives that
> some
> one had developed a better stemming algo for Solr than the built-in
> Porter
> stemming. Does anyone have link to that stemming module?
>
> Thanks,
>
> HH
>
>
>
>
>
RE: better stemming engine than Porter?
Posted by "Wagner,Harry" <wa...@oclc.org>.
Hi HH,
Here's a note I sent Solr-dev a while back:
---
I've implemented a Solr plug-in that wraps KStem for Solr use (someone
else had already written a Lucene wrapper for it). KStem is considered
to be more appropriate for library usage since it is much less
aggressive than Porter (i.e., searches for organization do NOT match on
organ!). If there is any interest in feeding this back into Solr I would
be happy to contribute it.
---
I believe there was interest in it, but I never opened an issue for it
and I don't know if it was ever followed-up on. I'd be happy to do that
now. Can someone on the Solr-dev team point me in the right direction
for opening an issue?
Thanks... harry
-----Original Message-----
From: Hung Huynh [mailto:hh@vinaconsulting.com]
Sent: Monday, April 21, 2008 11:59 AM
To: solr-user@lucene.apache.org
Subject: better stemming engine than Porter?
I recall I've read some where in one of the mailing-list archives that
some
one had developed a better stemming algo for Solr than the built-in
Porter
stemming. Does anyone have link to that stemming module?
Thanks,
HH
better stemming engine than Porter?
Posted by Hung Huynh <hh...@vinaconsulting.com>.
I recall I've read some where in one of the mailing-list archives that some
one had developed a better stemming algo for Solr than the built-in Porter
stemming. Does anyone have link to that stemming module?
Thanks,
HH
RE: How to troubleshoot this HTTP ERROR: 500 (NULL) error?
Posted by Hung Huynh <hh...@vinaconsulting.com>.
Thanks. I fixed the schema and it's working now.
Also my other problem of "not all defined fields showing up in the results"
is also resolved. I found out last night that Solr is case-sensitive. I
typed all fields in the schema in lower-case, and my CSV files had mixed
cases for some of the fields. These mixed case fields do not show up in the
XML results. Sorry, I've been working in the Windows environment for so long
and forgot about Unix-case-sensitive environment.
Thanks,
HH
-----Original Message-----
From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
Sent: Friday, April 18, 2008 4:44 PM
To: solr-user@lucene.apache.org
Subject: Re: How to troubleshoot this HTTP ERROR: 500 (NULL) error?
: java.lang.NullPointerException
:
: at
:
org.apache.solr.search.SolrQueryParser.getFieldQuery(SolrQueryParser.java:73
: )
I'm guessing this is one of the following issues...
http://issues.apache.org/jira/browse/SOLR-525
http://issues.apache.org/jira/browse/SOLR-529
...both have been changed in the trunk to provide better error messages,
but you can fix the root cause yourself (which you check your query/schema
to figure out which problem it is)
-Hoss
Re: How to troubleshoot this HTTP ERROR: 500 (NULL) error?
Posted by Chris Hostetter <ho...@fucit.org>.
: java.lang.NullPointerException
:
: at
: org.apache.solr.search.SolrQueryParser.getFieldQuery(SolrQueryParser.java:73
: )
I'm guessing this is one of the following issues...
http://issues.apache.org/jira/browse/SOLR-525
http://issues.apache.org/jira/browse/SOLR-529
...both have been changed in the trunk to provide better error messages,
but you can fix the root cause yourself (which you check your query/schema
to figure out which problem it is)
-Hoss
How to troubleshoot this HTTP ERROR: 500 (NULL) error?
Posted by Hung Huynh <hh...@vinaconsulting.com>.
I loaded about 20,000 docs. When I tried to do any search, I received the
following error. What should I do to fix this? Thanks.
--
HTTP ERROR: 500
null
java.lang.NullPointerException
at
org.apache.solr.search.SolrQueryParser.getFieldQuery(SolrQueryParser.java:73
)
at
org.apache.lucene.queryParser.QueryParser.Term(QueryParser.java:1092)
at
org.apache.lucene.queryParser.QueryParser.Clause(QueryParser.java:979)
at
org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:907)
at
org.apache.lucene.queryParser.QueryParser.TopLevelQuery(QueryParser.java:896
)
at
org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:146)
at
org.apache.solr.search.QueryParsing.parseQuery(QueryParsing.java:94)
at
org.apache.solr.request.StandardRequestHandler.handleRequestBody(StandardReq
uestHandler.java:109)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.
java:77)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:658)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:1
91)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
159)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1089)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerColl
ection.java:211)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:11
4)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnectio
n.java:821)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:22
6)
at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:4
42)
RequestURI=/solr/select/
Re: XSLT transform before update?
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Also, concatenation of values is also quite common. We should have a way of
doing this without forcing everybody to write code.
I think we should add a ConcatenateTransformer in DataImportHandler itself
which can take care of basic use-cases. A syntax like this may be good
enough:
<field column="myField" concatenate="field1, field2, field3,..."
separateBy=" " />
What do you think?
On Sat, Apr 19, 2008 at 12:41 AM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
> Hi David,
> Actually you can concatenate values, however you'll have to write a bit of
> code. You can write this in javascript (if you're using Java 6) or in Java.
>
> Basically, you need to write a Transformer to do it. Look at
> http://wiki.apache.org/solr/DataImportHandler#head-a6916b30b5d7605a990fb03c4ff461b3736496a9
>
> For example, lets say you get fields first-name and last-name in the XML.
> But in the schema.xml you have a field called "name" in which you need to
> concatenate the values of first-name and last-name (with a space in
> between). Create a Java class:
>
> public class ConcatenateTransformer { public Object
> transformRow(Map<String, Object> row) { String firstName =
> row.get("first-name"); String lastName = row.get("last-name");
> row.put("name", firstName + " " + lastName); return row; } }
>
> Add this class to solr's classpath by putting its jar in solr/WEB-INF/lib
>
> The data-config.xml should like this:
> <entity name="myEntity" processor="XPathEntityProcessor" url="
> http://myurl/example.xml"
> transformer="com.yourpackage.ConcatenateTransformer"> <field
> column="first-name" xpath="/record/first-name" /> <field column="last-name"
> xpath="/record/last-name" /> <field column="name" /> </entity>
>
> This will call ConcatenateTransformer.transformRow method for each row and
> you can concatenate any field with any field (or constant). Note that solr
> document will keep only those fields which are in the schema.xml, the rest
> are thrown away.
>
> If you don't want to write this in Java, you can use JavaScript by using
> the built-in ScriptTransformer, for an example look at
> http://wiki.apache.org/solr/DataImportHandler#head-27fcc2794bd71f7d727104ffc6b99e194bdb6ff9
>
> However, I'm beginning to realize that XSLT is a common need, let me see
> how best we can accomodate it in DataImportHandler. Which XSLT processor
> will you prefer?
>
> On Sat, Apr 19, 2008 at 12:13 AM, David Smiley @MITRE.org <
> DSMILEY@mitre.org> wrote:
>
> >
> > I'm in the same situation as you Daniel. The DataImportHandler is
> > pretty
> > awesome but I'd also prefer it had the power of XSLT. The XPath support
> > in
> > it doesn't suffice for me. And I can't do very basic things like
> > concatenate one value with another, say a constant even. It's too bad
> > there
> > isn't a mode that XSLT can be put in to to not build the whole file into
> > memory to do the transform. I've been looking into this and have turned
> > up
> > nothing. It would be neat if there was a STaX to multi-document
> > adapter, at
> > which point XSLT could be applied to the smaller fixed-size documents
> > instead of the entire data stream. I haven't found anything like this
> > so
> > it'd need to be built. For now my documents aren't too big to XSLT
> > in-memory.
> >
> > ~ David
> >
> >
> > Daniel Papasian wrote:
> > >
> > > Shalin Shekhar Mangar wrote:
> > >> Hi Daniel,
> > >>
> > >> Maybe if you can give us a sample of how your XML looks like, we can
> > >> suggest
> > >> how to use SOLR-469 (Data Import Handler) to index it. Most of the
> > >> use-cases
> > >> we have yet encountered are solvable using the XPathEntityProcessor
> > in
> > >> DataImportHandler without using XSLT, for details look at
> > >>
> > http://wiki.apache.org/solr/DataImportHandler#head-e68aa93c9ca7b8d261cede2bf1d6110ab1725476
> > >
> > > I think even if it is possible to use SOLR-469 for my needs, I'd still
> > > prefer the XSLT approach, because it's going to be a bit of
> > > configuration either way, and I'd rather it be an XSLT stylesheet than
> > > solrconfig.xml. In addition, I haven't yet decided whether I want to
> > > apply any patches to the version that we will deploy, but if I do go
> > > down the route of the XSLT transform patch, if I end up having to back
> > > it out the amount of work that it would be for me to do the transform
> > at
> > > the XML source would be negligible, where it would be quite a bit of
> > > work ahead of me to go from using the DataImportHandler to not using
> > it
> > > at all.
> > >
> > > Because both the solr instance and the XML source are in house, I have
> > > the ability to apply the XSLT at the source instead of at solr.
> > > However, there are different teams of people that control the XML
> > source
> > > and solr, so it would require a bit more office coordination to do it
> > on
> > > the backend.
> > >
> > > The data is a filemaker XML export (DTD fmresultset) and it looks
> > > roughly like this:
> > > <fmresultset>
> > > <resultset>
> > > <field name="ID"><data>125</data></field>
> > > <field name="organization"><data>Ford Foundation</data></field>
> > > ...
> > > <relatedset table="Employees">
> > > <record>
> > > <field name="ID"><data>Y5-A</data></field>
> > > <field name="Name"><data>John Smith</data></field>
> > > </record>
> > > <record>
> > > <field name="ID"><data>Y5-B</data></field>
> > > <field name="Name"><data>Jane Doe</data></field>
> > > </record>
> > > </relatedset>
> > > </fmresultset>
> > >
> > > I'm taking the product of the resultset and the relatedset, using both
> > > IDs concatenated as a unique identifier, like so:
> > >
> > > <doc>
> > > <field name="ID">125Y5-A</field>
> > > <field name="organization">Ford Foundation</field>
> > > <field name="Name">John Smith</field>
> > > </doc>
> > > <doc>
> > > <field name="ID">125Y5-B</field>
> > > <field name="organization">Ford Foundation</field>
> > > <field name="Name">Jane Doe</field>
> > > </doc>
> > >
> > > I can do the transform pretty simply with XSLT. I suppose it is
> > > possible to get the DataImportHandler to do this, but I'm not yet
> > > convinced that it's easier.
> > >
> > > Daniel
> > >
> > >
> >
> > --
> > View this message in context:
> > http://www.nabble.com/XSLT-transform-before-update--tp16738227p16764009.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
--
Regards,
Shalin Shekhar Mangar.
Re: XSLT transform before update?
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Hi David,
Actually you can concatenate values, however you'll have to write a bit of
code. You can write this in javascript (if you're using Java 6) or in Java.
Basically, you need to write a Transformer to do it. Look at
http://wiki.apache.org/solr/DataImportHandler#head-a6916b30b5d7605a990fb03c4ff461b3736496a9
For example, lets say you get fields first-name and last-name in the XML.
But in the schema.xml you have a field called "name" in which you need to
concatenate the values of first-name and last-name (with a space in
between). Create a Java class:
public class ConcatenateTransformer { public Object transformRow(Map<String,
Object> row) { String firstName = row.get("first-name"); String lastName =
row.get("last-name"); row.put("name", firstName + " " + lastName); return
row; } }
Add this class to solr's classpath by putting its jar in solr/WEB-INF/lib
The data-config.xml should like this:
<entity name="myEntity" processor="XPathEntityProcessor" url="
http://myurl/example.xml"
transformer="com.yourpackage.ConcatenateTransformer"> <field
column="first-name" xpath="/record/first-name" /> <field column="last-name"
xpath="/record/last-name" /> <field column="name" /> </entity>
This will call ConcatenateTransformer.transformRow method for each row and
you can concatenate any field with any field (or constant). Note that solr
document will keep only those fields which are in the schema.xml, the rest
are thrown away.
If you don't want to write this in Java, you can use JavaScript by using the
built-in ScriptTransformer, for an example look at
http://wiki.apache.org/solr/DataImportHandler#head-27fcc2794bd71f7d727104ffc6b99e194bdb6ff9
However, I'm beginning to realize that XSLT is a common need, let me see how
best we can accomodate it in DataImportHandler. Which XSLT processor will
you prefer?
On Sat, Apr 19, 2008 at 12:13 AM, David Smiley @MITRE.org <DS...@mitre.org>
wrote:
>
> I'm in the same situation as you Daniel. The DataImportHandler is pretty
> awesome but I'd also prefer it had the power of XSLT. The XPath support
> in
> it doesn't suffice for me. And I can't do very basic things like
> concatenate one value with another, say a constant even. It's too bad
> there
> isn't a mode that XSLT can be put in to to not build the whole file into
> memory to do the transform. I've been looking into this and have turned
> up
> nothing. It would be neat if there was a STaX to multi-document adapter,
> at
> which point XSLT could be applied to the smaller fixed-size documents
> instead of the entire data stream. I haven't found anything like this so
> it'd need to be built. For now my documents aren't too big to XSLT
> in-memory.
>
> ~ David
>
>
> Daniel Papasian wrote:
> >
> > Shalin Shekhar Mangar wrote:
> >> Hi Daniel,
> >>
> >> Maybe if you can give us a sample of how your XML looks like, we can
> >> suggest
> >> how to use SOLR-469 (Data Import Handler) to index it. Most of the
> >> use-cases
> >> we have yet encountered are solvable using the XPathEntityProcessor in
> >> DataImportHandler without using XSLT, for details look at
> >>
> http://wiki.apache.org/solr/DataImportHandler#head-e68aa93c9ca7b8d261cede2bf1d6110ab1725476
> >
> > I think even if it is possible to use SOLR-469 for my needs, I'd still
> > prefer the XSLT approach, because it's going to be a bit of
> > configuration either way, and I'd rather it be an XSLT stylesheet than
> > solrconfig.xml. In addition, I haven't yet decided whether I want to
> > apply any patches to the version that we will deploy, but if I do go
> > down the route of the XSLT transform patch, if I end up having to back
> > it out the amount of work that it would be for me to do the transform at
> > the XML source would be negligible, where it would be quite a bit of
> > work ahead of me to go from using the DataImportHandler to not using it
> > at all.
> >
> > Because both the solr instance and the XML source are in house, I have
> > the ability to apply the XSLT at the source instead of at solr.
> > However, there are different teams of people that control the XML source
> > and solr, so it would require a bit more office coordination to do it on
> > the backend.
> >
> > The data is a filemaker XML export (DTD fmresultset) and it looks
> > roughly like this:
> > <fmresultset>
> > <resultset>
> > <field name="ID"><data>125</data></field>
> > <field name="organization"><data>Ford Foundation</data></field>
> > ...
> > <relatedset table="Employees">
> > <record>
> > <field name="ID"><data>Y5-A</data></field>
> > <field name="Name"><data>John Smith</data></field>
> > </record>
> > <record>
> > <field name="ID"><data>Y5-B</data></field>
> > <field name="Name"><data>Jane Doe</data></field>
> > </record>
> > </relatedset>
> > </fmresultset>
> >
> > I'm taking the product of the resultset and the relatedset, using both
> > IDs concatenated as a unique identifier, like so:
> >
> > <doc>
> > <field name="ID">125Y5-A</field>
> > <field name="organization">Ford Foundation</field>
> > <field name="Name">John Smith</field>
> > </doc>
> > <doc>
> > <field name="ID">125Y5-B</field>
> > <field name="organization">Ford Foundation</field>
> > <field name="Name">Jane Doe</field>
> > </doc>
> >
> > I can do the transform pretty simply with XSLT. I suppose it is
> > possible to get the DataImportHandler to do this, but I'm not yet
> > convinced that it's easier.
> >
> > Daniel
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/XSLT-transform-before-update--tp16738227p16764009.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: XSLT transform before update?
Posted by "David Smiley @MITRE.org" <DS...@mitre.org>.
I'm in the same situation as you Daniel. The DataImportHandler is pretty
awesome but I'd also prefer it had the power of XSLT. The XPath support in
it doesn't suffice for me. And I can't do very basic things like
concatenate one value with another, say a constant even. It's too bad there
isn't a mode that XSLT can be put in to to not build the whole file into
memory to do the transform. I've been looking into this and have turned up
nothing. It would be neat if there was a STaX to multi-document adapter, at
which point XSLT could be applied to the smaller fixed-size documents
instead of the entire data stream. I haven't found anything like this so
it'd need to be built. For now my documents aren't too big to XSLT
in-memory.
~ David
Daniel Papasian wrote:
>
> Shalin Shekhar Mangar wrote:
>> Hi Daniel,
>>
>> Maybe if you can give us a sample of how your XML looks like, we can
>> suggest
>> how to use SOLR-469 (Data Import Handler) to index it. Most of the
>> use-cases
>> we have yet encountered are solvable using the XPathEntityProcessor in
>> DataImportHandler without using XSLT, for details look at
>> http://wiki.apache.org/solr/DataImportHandler#head-e68aa93c9ca7b8d261cede2bf1d6110ab1725476
>
> I think even if it is possible to use SOLR-469 for my needs, I'd still
> prefer the XSLT approach, because it's going to be a bit of
> configuration either way, and I'd rather it be an XSLT stylesheet than
> solrconfig.xml. In addition, I haven't yet decided whether I want to
> apply any patches to the version that we will deploy, but if I do go
> down the route of the XSLT transform patch, if I end up having to back
> it out the amount of work that it would be for me to do the transform at
> the XML source would be negligible, where it would be quite a bit of
> work ahead of me to go from using the DataImportHandler to not using it
> at all.
>
> Because both the solr instance and the XML source are in house, I have
> the ability to apply the XSLT at the source instead of at solr.
> However, there are different teams of people that control the XML source
> and solr, so it would require a bit more office coordination to do it on
> the backend.
>
> The data is a filemaker XML export (DTD fmresultset) and it looks
> roughly like this:
> <fmresultset>
> <resultset>
> <field name="ID"><data>125</data></field>
> <field name="organization"><data>Ford Foundation</data></field>
> ...
> <relatedset table="Employees">
> <record>
> <field name="ID"><data>Y5-A</data></field>
> <field name="Name"><data>John Smith</data></field>
> </record>
> <record>
> <field name="ID"><data>Y5-B</data></field>
> <field name="Name"><data>Jane Doe</data></field>
> </record>
> </relatedset>
> </fmresultset>
>
> I'm taking the product of the resultset and the relatedset, using both
> IDs concatenated as a unique identifier, like so:
>
> <doc>
> <field name="ID">125Y5-A</field>
> <field name="organization">Ford Foundation</field>
> <field name="Name">John Smith</field>
> </doc>
> <doc>
> <field name="ID">125Y5-B</field>
> <field name="organization">Ford Foundation</field>
> <field name="Name">Jane Doe</field>
> </doc>
>
> I can do the transform pretty simply with XSLT. I suppose it is
> possible to get the DataImportHandler to do this, but I'm not yet
> convinced that it's easier.
>
> Daniel
>
>
--
View this message in context: http://www.nabble.com/XSLT-transform-before-update--tp16738227p16764009.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: XSLT transform before update?
Posted by Daniel Papasian <da...@chronicle.com>.
Shalin Shekhar Mangar wrote:
> Hi Daniel,
>
> Maybe if you can give us a sample of how your XML looks like, we can suggest
> how to use SOLR-469 (Data Import Handler) to index it. Most of the use-cases
> we have yet encountered are solvable using the XPathEntityProcessor in
> DataImportHandler without using XSLT, for details look at
> http://wiki.apache.org/solr/DataImportHandler#head-e68aa93c9ca7b8d261cede2bf1d6110ab1725476
I think even if it is possible to use SOLR-469 for my needs, I'd still
prefer the XSLT approach, because it's going to be a bit of
configuration either way, and I'd rather it be an XSLT stylesheet than
solrconfig.xml. In addition, I haven't yet decided whether I want to
apply any patches to the version that we will deploy, but if I do go
down the route of the XSLT transform patch, if I end up having to back
it out the amount of work that it would be for me to do the transform at
the XML source would be negligible, where it would be quite a bit of
work ahead of me to go from using the DataImportHandler to not using it
at all.
Because both the solr instance and the XML source are in house, I have
the ability to apply the XSLT at the source instead of at solr.
However, there are different teams of people that control the XML source
and solr, so it would require a bit more office coordination to do it on
the backend.
The data is a filemaker XML export (DTD fmresultset) and it looks
roughly like this:
<fmresultset>
<resultset>
<field name="ID"><data>125</data></field>
<field name="organization"><data>Ford Foundation</data></field>
...
<relatedset table="Employees">
<record>
<field name="ID"><data>Y5-A</data></field>
<field name="Name"><data>John Smith</data></field>
</record>
<record>
<field name="ID"><data>Y5-B</data></field>
<field name="Name"><data>Jane Doe</data></field>
</record>
</relatedset>
</fmresultset>
I'm taking the product of the resultset and the relatedset, using both
IDs concatenated as a unique identifier, like so:
<doc>
<field name="ID">125Y5-A</field>
<field name="organization">Ford Foundation</field>
<field name="Name">John Smith</field>
</doc>
<doc>
<field name="ID">125Y5-B</field>
<field name="organization">Ford Foundation</field>
<field name="Name">Jane Doe</field>
</doc>
I can do the transform pretty simply with XSLT. I suppose it is
possible to get the DataImportHandler to do this, but I'm not yet
convinced that it's easier.
Daniel
Re: XSLT transform before update?
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Hi Daniel,
Maybe if you can give us a sample of how your XML looks like, we can suggest
how to use SOLR-469 (Data Import Handler) to index it. Most of the use-cases
we have yet encountered are solvable using the XPathEntityProcessor in
DataImportHandler without using XSLT, for details look at
http://wiki.apache.org/solr/DataImportHandler#head-e68aa93c9ca7b8d261cede2bf1d6110ab1725476
If you're willing to write code, you can do almost anything with
DataImportHandler. If this is a general need, I can look into adding XSLT
support in Data Import Handler.
On Thu, Apr 17, 2008 at 9:13 AM, Daniel Papasian <
Daniel.Papasian@chronicle.com> wrote:
> Hey everyone,
>
> I'm experimenting with updating solr from a remote XML source, using an
> XSLT transform to get it into the solr XML syntax (and yes, I've looked
> into SOLR-469, but disregarded it as I need to do quite a bit using XSLT
> to get it to what I can index) to let me maintain an index.
>
> I'm looking at using stream.url, but I need to do the XSLT at some point
> in there. I would prefer to do the XSLT on the client (solr) side of
> the transfer, for various reasons.
>
> Is there a way to implement a custom request handler or similar to get
> solr to apply an XSLT transform to the content stream before it attempts
> to parse it? If not possible OOTB, where would be the right place to
> add said functionality?
>
> Thanks much for your help,
>
> Daniel
>
--
Regards,
Shalin Shekhar Mangar.
Re: XSLT transform before update?
Posted by Chris Hostetter <ho...@fucit.org>.
: Is there a way to implement a custom request handler or similar to get
: solr to apply an XSLT transform to the content stream before it attempts
: to parse it? If not possible OOTB, where would be the right place to
: add said functionality?
take a look at SOLR-285 and SOLR-370 ... a RequestHandler is the right way
to go, the biggest problems with the patch in SOLR-285 at the moment are:
a) i wrote it and i don't know much about doing XSLT transformations in
java efficiently.
b) the existing XSLT Transformer "caching" code in Solr is really
trivial and not suitable for any real volume ... if it were overhauled to
take advantage of the standard SolrCache APIs it would be a lot more
reusable by both the XsltResponseWriter and a new XsltUpdateHandler.
-Hoss