You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by Erik Hatcher <er...@ehatchersolutions.com> on 2009/07/29 21:53:08 UTC
DataImportHandler, XPathEntityProcessor, $hasMore, infinite loop
I've been troubleshooting an issue where we're trying to load
documents through DIH's URLDataSource and XPathEntityProcessor, where
we want to leverage the $hasMore feature to request to a new URL.
I've been tinkering with this using a very simple example, two XML
files -
solr.xml:
<add>
<doc>
<field name="id">SOLR1000</field>
</doc>
<doc>
<field name="id">**HASMORE**</field>
</doc>
</add>
solr2.xml
<add>
<doc>
<field name="id">SOLR2k</field>
</doc>
</add>
My DIH config is:
<?xml version="1.0"?>
<dataConfig>
<dataSource type="URLDataSource" baseUrl="file:///Users/erikhatcher/dev/solr/example/exampledocs/
"
readTimeout="180000" connectionTimeout="60000"/>
<script>
<![CDATA[
function checkForMore(row, context) {
print("### checkForMore: " + row);
if (row.get('id') == '**HASMORE**') {
print("#### hasMore ####");
row.put('$hasMore', 'true');
row.put('$nextUrl', 'file:///Users/erikhatcher/dev/solr/example/exampledocs/solr2.xml')
;
row.put('$skipRow', 'true');
} else {
row.put('$hasMore', 'false');
}
return row;
}
]]>
</script>
<document name="docs">
<entity name="doc"
processor="XPathEntityProcessor"
url="solr.xml"
forEach="/add/doc"
stream="true"
transformer
="DateFormatTransformer,TemplateTransformer,script:checkForMore"
onError="abort">
<field column="id" xpath="/add/doc/field[@name='id']"/>
</entity>
</document>
</dataConfig>
Without the else clause in checkForMore to set $hasMore to false, an
infinite loop occurs and solr2.xml is requested repeatedly. This is
because once $hasMore is set on a row,
XPathEntityProcess#readUsefulVars sets it in entity scope and it never
gets unset. Is this intentional? Shouldn't $hasMore get reset after
more is requested?
On a related note, it would seem useful to allow $hasMore/$skipRow/
$nextUrl to be controlled from the XML data rather than solely from a
transformer. But $prefixed fields are ignored by DIH, right?
I'm still looking for that holy grail of a good example leveraging
$hasMore/$nextUrl! :)
Thanks,
Erik
Re: DataImportHandler, XPathEntityProcessor, $hasMore, infinite loop
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
2009/7/30 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>:
> On Thu, Jul 30, 2009 at 1:23 AM, Erik Hatcher<er...@ehatchersolutions.com> wrote:
>> I've been troubleshooting an issue where we're trying to load documents
>> through DIH's URLDataSource and XPathEntityProcessor, where we want to
>> leverage the $hasMore feature to request to a new URL.
>>
>> I've been tinkering with this using a very simple example, two XML files -
>>
>> solr.xml:
>> <add>
>> <doc>
>> <field name="id">SOLR1000</field>
>> </doc>
>> <doc>
>> <field name="id">**HASMORE**</field>
>> </doc>
>> </add>
>>
>> solr2.xml
>> <add>
>> <doc>
>> <field name="id">SOLR2k</field>
>> </doc>
>> </add>
>>
>> My DIH config is:
>>
>> <?xml version="1.0"?>
>> <dataConfig>
>> <dataSource type="URLDataSource"
>> baseUrl="file:///Users/erikhatcher/dev/solr/example/exampledocs/"
>> readTimeout="180000" connectionTimeout="60000"/>
>>
>> <script>
>> <![CDATA[
>> function checkForMore(row, context) {
>> print("### checkForMore: " + row);
>> if (row.get('id') == '**HASMORE**') {
>> print("#### hasMore ####");
>> row.put('$hasMore', 'true');
>> row.put('$nextUrl',
>> 'file:///Users/erikhatcher/dev/solr/example/exampledocs/solr2.xml');
>> row.put('$skipRow', 'true');
>> } else {
>> row.put('$hasMore', 'false');
>> }
>> return row;
>> }
>> ]]>
>> </script>
>>
>> <document name="docs">
>> <entity name="doc"
>> processor="XPathEntityProcessor"
>> url="solr.xml"
>> forEach="/add/doc"
>> stream="true"
>>
>> transformer="DateFormatTransformer,TemplateTransformer,script:checkForMore"
>> onError="abort">
>> <field column="id" xpath="/add/doc/field[@name='id']"/>
>> </entity>
>> </document>
>> </dataConfig>
>>
>> Without the else clause in checkForMore to set $hasMore to false, an
>> infinite loop occurs and solr2.xml is requested repeatedly. This is because
>> once $hasMore is set on a row, XPathEntityProcess#readUsefulVars sets it in
>> entity scope and it never gets unset. Is this intentional? Shouldn't
>> $hasMore get reset after more is requested?
>
> I would say we must reset it after using once.
>>
>> On a related note, it would seem useful to allow $hasMore/$skipRow/$nextUrl
>> to be controlled from the XML data rather than solely from a transformer.
>> But $prefixed fields are ignored by DIH, right?
> This is possible using a RegexTransformer (so you may not need to
> write your own)
>
> <field column="$hasMore" regex="HASMORE" replaceWith="true"/>
a small correction
<field column="$hasMore" regex="HASMORE" replaceWith="true" sourceColName="id"/>
>
>
>>
>> I'm still looking for that holy grail of a good example leveraging
>> $hasMore/$nextUrl! :)
>>
>> Thanks,
>> Erik
>>
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: DataImportHandler, XPathEntityProcessor, $hasMore, infinite loop
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
On Thu, Jul 30, 2009 at 1:23 AM, Erik Hatcher<er...@ehatchersolutions.com> wrote:
> I've been troubleshooting an issue where we're trying to load documents
> through DIH's URLDataSource and XPathEntityProcessor, where we want to
> leverage the $hasMore feature to request to a new URL.
>
> I've been tinkering with this using a very simple example, two XML files -
>
> solr.xml:
> <add>
> <doc>
> <field name="id">SOLR1000</field>
> </doc>
> <doc>
> <field name="id">**HASMORE**</field>
> </doc>
> </add>
>
> solr2.xml
> <add>
> <doc>
> <field name="id">SOLR2k</field>
> </doc>
> </add>
>
> My DIH config is:
>
> <?xml version="1.0"?>
> <dataConfig>
> <dataSource type="URLDataSource"
> baseUrl="file:///Users/erikhatcher/dev/solr/example/exampledocs/"
> readTimeout="180000" connectionTimeout="60000"/>
>
> <script>
> <![CDATA[
> function checkForMore(row, context) {
> print("### checkForMore: " + row);
> if (row.get('id') == '**HASMORE**') {
> print("#### hasMore ####");
> row.put('$hasMore', 'true');
> row.put('$nextUrl',
> 'file:///Users/erikhatcher/dev/solr/example/exampledocs/solr2.xml');
> row.put('$skipRow', 'true');
> } else {
> row.put('$hasMore', 'false');
> }
> return row;
> }
> ]]>
> </script>
>
> <document name="docs">
> <entity name="doc"
> processor="XPathEntityProcessor"
> url="solr.xml"
> forEach="/add/doc"
> stream="true"
>
> transformer="DateFormatTransformer,TemplateTransformer,script:checkForMore"
> onError="abort">
> <field column="id" xpath="/add/doc/field[@name='id']"/>
> </entity>
> </document>
> </dataConfig>
>
> Without the else clause in checkForMore to set $hasMore to false, an
> infinite loop occurs and solr2.xml is requested repeatedly. This is because
> once $hasMore is set on a row, XPathEntityProcess#readUsefulVars sets it in
> entity scope and it never gets unset. Is this intentional? Shouldn't
> $hasMore get reset after more is requested?
I would say we must reset it after using once.
>
> On a related note, it would seem useful to allow $hasMore/$skipRow/$nextUrl
> to be controlled from the XML data rather than solely from a transformer.
> But $prefixed fields are ignored by DIH, right?
This is possible using a RegexTransformer (so you may not need to
write your own)
<field column="$hasMore" regex="HASMORE" replaceWith="true"/>
>
> I'm still looking for that holy grail of a good example leveraging
> $hasMore/$nextUrl! :)
>
> Thanks,
> Erik
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com