You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mark Ellul <ma...@catalystic.com> on 2009/11/11 16:49:39 UTC
${dataimporter.delta.twitter_id} not getting populated in
deltaImportQuery
Hi,
I have a interesting issue...
Basically I am trying to delta imports on solr 1.4 on a postgresql 8.3
database.
Basically when I am running a delta import with the entity below I get an
exception (see below the entity definition) showing the query its trying to
run and you can see that its not populating the where clause of my
dataImportQuery.
I have tried ${dataimporter.delta.twitter_id} and ${dataimporter.delta.id}
and get the same exceptions.
Am I missing something obvious?
Any help would be appreciated!
Regards
Mark
<entity name="Tweeter" pk="twitter_id"
query="
select twitter_id,
twitter_id as pk,
1 as site_id,
screen_name
from api_tweeter WHERE
tweet_mapreduce_on IS NOT NULL;
" transformer="TemplateTransformer"
deltaImportQuery="
select twitter_id,
twitter_id as pk,
1 as site_id,
screen_name
from api_tweeter
where twitter_id=${dataimporter.delta.twitter_id };
"
deltaQuery ="select twitter_id from api_tweeter where modified_on >
'${dataimporter.last_index_time}' and tweet_mapreduce_on IS NOT NULL;"
>
<field name="twitter_id" column="twitter_id" />
</entity>
INFO: Completed parentDeltaQuery for Entity: Tweeter
Nov 11, 2009 3:35:44 PM org.apache.solr.handler.dataimport.DocBuilder
buildDocument
SEVERE: Exception while processing: Tweeter document : SolrInputDocument[{}]
org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
execute query: select twitter_id, twitter_id
as pk, 1 as site_id, screen_name from api_tweeter where
twitter_id=; Processing Document # 1
at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
at
org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
at
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at end of
input
Position: 1197
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:345)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:246)
... 11 more
Nov 11, 2009 3:35:44 PM org.apache.solr.handler.dataimport.DataImporter
doDeltaImport
SEVERE: Delta Import Failed
org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
execute query: select twitter_id, twitter_id
as pk, 1 as site_id, screen_name from api_tweeter where
twitter_id=; Processing Document # 1
at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
at
org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
at
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at end of
input
Position: 1197
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:345)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:246)
... 11 more
Nov 11, 2009 3:35:44 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: start rollback
Nov 11, 2009 3:35:44 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: end_rollback
Fwd: ${dataimporter.delta.twitter_id} not getting populated in
deltaImportQuery
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
---------- Forwarded message ----------
From: Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
Date: 2009/11/12
Subject: Re: ${dataimporter.delta.twitter_id} not getting populated in
deltaImportQuery
To: Mark Ellul <ma...@catalystic.com>
On Thu, Nov 12, 2009 at 8:17 PM, Mark Ellul <ma...@catalystic.com> wrote:
> I think I got it working, thanks for your response... Once I removed the
> TemplateTransformer from the entity.
> Could that have been the issue? Could the template transformer have been
> changing the ${dataimporter.delta.twitter_id} into nothing?
But the though templateTransformer is mentioned , it is not applied on
any field . is it? I do not see the attribute 'template' on any field.
> Regards
> Mark
>
> 2009/11/12 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
>>
>> delta-import is slightly tricky ,. There are no traps which will let
>> you know the intermediate data. That is why I suggested you to do a
>> delta-import using the full-import .It can probably reveal what is the
>> problem
>>
>> On Thu, Nov 12, 2009 at 6:05 PM, Mark Ellul <ma...@catalystic.com> wrote:
>> > Hi Noble,
>> > Thanks for the response.
>> > CAPS is not the issue.
>> > Can you please confirm the link below is the code for the
>> > SQLEntityProcessor
>> > in the release 1.4?
>> >
>> > http://svn.apache.org/viewvc/lucene/solr/tags/release-1.4.0/contrib/dataimporthandler/src/main/java/org/apache/solr/handler/dataimport/SqlEntityProcessor.java?revision=834197&view=markup
>> > Is there a way to output what is returned from the deltaQuery?
>> > Or the actual queries sent to the database server?
>> > Regards
>> > Mark
>> > 2009/11/12 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
>> >>
>> >> are you sure the data comes back in the same name. Some DBs return the
>> >> field names in ALL CAPS
>> >>
>> >> you may try out a delta_import using a full import too
>> >>
>> >>
>> >>
>> >> http://wiki.apache.org/solr/DataImportHandlerFaq#My_delta-import_goes_out_of_memory_._Any_workaround_.3F
>> >>
>> >> On Wed, Nov 11, 2009 at 9:55 PM, Mark Ellul <ma...@catalystic.com>
>> >> wrote:
>> >> > I have 2 entities from the root node, not sure if that makes a
>> >> > difference!
>> >> >
>> >> > On Wed, Nov 11, 2009 at 4:49 PM, Mark Ellul <ma...@catalystic.com>
>> >> > wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> I have a interesting issue...
>> >> >>
>> >> >> Basically I am trying to delta imports on solr 1.4 on a postgresql
>> >> >> 8.3
>> >> >> database.
>> >> >>
>> >> >> Basically when I am running a delta import with the entity below I
>> >> >> get
>> >> >> an
>> >> >> exception (see below the entity definition) showing the query its
>> >> >> trying to
>> >> >> run and you can see that its not populating the where clause of my
>> >> >> dataImportQuery.
>> >> >>
>> >> >> I have tried ${dataimporter.delta.twitter_id} and
>> >> >> ${dataimporter.delta.id}
>> >> >> and get the same exceptions.
>> >> >>
>> >> >> Am I missing something obvious?
>> >> >>
>> >> >> Any help would be appreciated!
>> >> >>
>> >> >> Regards
>> >> >>
>> >> >> Mark
>> >> >>
>> >> >>
>> >> >> <entity name="Tweeter" pk="twitter_id"
>> >> >> query="
>> >> >> select twitter_id,
>> >> >> twitter_id as pk,
>> >> >> 1 as site_id,
>> >> >> screen_name
>> >> >>
>> >> >> from api_tweeter WHERE
>> >> >> tweet_mapreduce_on IS NOT NULL;
>> >> >> " transformer="TemplateTransformer"
>> >> >>
>> >> >> deltaImportQuery="
>> >> >> select twitter_id,
>> >> >> twitter_id as pk,
>> >> >> 1 as site_id,
>> >> >> screen_name
>> >> >>
>> >> >> from api_tweeter
>> >> >> where twitter_id=${dataimporter.delta.twitter_id };
>> >> >> "
>> >> >> deltaQuery ="select twitter_id from api_tweeter where modified_on >
>> >> >> '${dataimporter.last_index_time}' and tweet_mapreduce_on IS NOT
>> >> >> NULL;"
>> >> >>
>> >> >> >
>> >> >> <field name="twitter_id" column="twitter_id" />
>> >> >>
>> >> >>
>> >> >> </entity>
>> >> >>
>> >> >>
>> >> >> INFO: Completed parentDeltaQuery for Entity: Tweeter
>> >> >> Nov 11, 2009 3:35:44 PM
>> >> >> org.apache.solr.handler.dataimport.DocBuilder
>> >> >> buildDocument
>> >> >> SEVERE: Exception while processing: Tweeter document :
>> >> >> SolrInputDocument[{}]
>> >> >> org.apache.solr.handler.dataimport.DataImportHandlerException:
>> >> >> Unable
>> >> >> to
>> >> >> execute query: select twitter_id,
>> >> >> twitter_id
>> >> >> as pk, 1 as site_id, screen_name from api_tweeter
>> >> >> where
>> >> >> twitter_id=; Processing Document # 1
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>> >> >> Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at
>> >> >> end of
>> >> >> input
>> >> >> Position: 1197
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:345)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:246)
>> >> >> ... 11 more
>> >> >> Nov 11, 2009 3:35:44 PM
>> >> >> org.apache.solr.handler.dataimport.DataImporter
>> >> >> doDeltaImport
>> >> >> SEVERE: Delta Import Failed
>> >> >> org.apache.solr.handler.dataimport.DataImportHandlerException:
>> >> >> Unable
>> >> >> to
>> >> >> execute query: select twitter_id,
>> >> >> twitter_id
>> >> >> as pk, 1 as site_id, screen_name from api_tweeter
>> >> >> where
>> >> >> twitter_id=; Processing Document # 1
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>> >> >> Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at
>> >> >> end of
>> >> >> input
>> >> >> Position: 1197
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:345)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:246)
>> >> >> ... 11 more
>> >> >> Nov 11, 2009 3:35:44 PM org.apache.solr.update.DirectUpdateHandler2
>> >> >> rollback
>> >> >> INFO: start rollback
>> >> >> Nov 11, 2009 3:35:44 PM org.apache.solr.update.DirectUpdateHandler2
>> >> >> rollback
>> >> >> INFO: end_rollback
>> >> >>
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> -----------------------------------------------------
>> >> Noble Paul | Principal Engineer| AOL | http://aol.com
>> >
>> >
>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul | Principal Engineer| AOL | http://aol.com
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: ${dataimporter.delta.twitter_id} not getting populated in
deltaImportQuery
Posted by Mark Ellul <ma...@catalystic.com>.
Hi Noble,
Thanks for the response.
CAPS is not the issue.
Can you please confirm the link below is the code for the SQLEntityProcessor
in the release 1.4?
http://svn.apache.org/viewvc/lucene/solr/tags/release-1.4.0/contrib/dataimporthandler/src/main/java/org/apache/solr/handler/dataimport/SqlEntityProcessor.java?revision=834197&view=markup
Is there a way to output what is returned from the deltaQuery?
Or the actual queries sent to the database server?
Regards
Mark
2009/11/12 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
> are you sure the data comes back in the same name. Some DBs return the
> field names in ALL CAPS
>
> you may try out a delta_import using a full import too
>
>
> http://wiki.apache.org/solr/DataImportHandlerFaq#My_delta-import_goes_out_of_memory_._Any_workaround_.3F
>
> On Wed, Nov 11, 2009 at 9:55 PM, Mark Ellul <ma...@catalystic.com> wrote:
> > I have 2 entities from the root node, not sure if that makes a
> difference!
> >
> > On Wed, Nov 11, 2009 at 4:49 PM, Mark Ellul <ma...@catalystic.com> wrote:
> >
> >> Hi,
> >>
> >> I have a interesting issue...
> >>
> >> Basically I am trying to delta imports on solr 1.4 on a postgresql 8.3
> >> database.
> >>
> >> Basically when I am running a delta import with the entity below I get
> an
> >> exception (see below the entity definition) showing the query its
> trying to
> >> run and you can see that its not populating the where clause of my
> >> dataImportQuery.
> >>
> >> I have tried ${dataimporter.delta.twitter_id} and ${
> dataimporter.delta.id}
> >> and get the same exceptions.
> >>
> >> Am I missing something obvious?
> >>
> >> Any help would be appreciated!
> >>
> >> Regards
> >>
> >> Mark
> >>
> >>
> >> <entity name="Tweeter" pk="twitter_id"
> >> query="
> >> select twitter_id,
> >> twitter_id as pk,
> >> 1 as site_id,
> >> screen_name
> >>
> >> from api_tweeter WHERE
> >> tweet_mapreduce_on IS NOT NULL;
> >> " transformer="TemplateTransformer"
> >>
> >> deltaImportQuery="
> >> select twitter_id,
> >> twitter_id as pk,
> >> 1 as site_id,
> >> screen_name
> >>
> >> from api_tweeter
> >> where twitter_id=${dataimporter.delta.twitter_id };
> >> "
> >> deltaQuery ="select twitter_id from api_tweeter where modified_on >
> >> '${dataimporter.last_index_time}' and tweet_mapreduce_on IS NOT NULL;"
> >>
> >> >
> >> <field name="twitter_id" column="twitter_id" />
> >>
> >>
> >> </entity>
> >>
> >>
> >> INFO: Completed parentDeltaQuery for Entity: Tweeter
> >> Nov 11, 2009 3:35:44 PM org.apache.solr.handler.dataimport.DocBuilder
> >> buildDocument
> >> SEVERE: Exception while processing: Tweeter document :
> >> SolrInputDocument[{}]
> >> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
> >> execute query: select twitter_id,
> twitter_id
> >> as pk, 1 as site_id, screen_name from api_tweeter
> where
> >> twitter_id=; Processing Document # 1
> >> at
> >>
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
> >> at
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
> >> at
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
> >> at
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
> >> at
> >>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
> >> at
> >>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
> >> at
> >>
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
> >> at
> >>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
> >> at
> >>
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
> >> at
> >>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
> >> at
> >>
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
> >> at
> >>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
> >> at
> >>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
> >> Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at end
> of
> >> input
> >> Position: 1197
> >> at
> >>
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
> >> at
> >>
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
> >> at
> >>
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> >> at
> >>
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
> >> at
> >>
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
> >> at
> >>
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:345)
> >> at
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:246)
> >> ... 11 more
> >> Nov 11, 2009 3:35:44 PM org.apache.solr.handler.dataimport.DataImporter
> >> doDeltaImport
> >> SEVERE: Delta Import Failed
> >> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
> >> execute query: select twitter_id,
> twitter_id
> >> as pk, 1 as site_id, screen_name from api_tweeter
> where
> >> twitter_id=; Processing Document # 1
> >> at
> >>
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
> >> at
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
> >> at
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
> >> at
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
> >> at
> >>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
> >> at
> >>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
> >> at
> >>
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
> >> at
> >>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
> >> at
> >>
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
> >> at
> >>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
> >> at
> >>
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
> >> at
> >>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
> >> at
> >>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
> >> Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at end
> of
> >> input
> >> Position: 1197
> >> at
> >>
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
> >> at
> >>
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
> >> at
> >>
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> >> at
> >>
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
> >> at
> >>
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
> >> at
> >>
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:345)
> >> at
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:246)
> >> ... 11 more
> >> Nov 11, 2009 3:35:44 PM org.apache.solr.update.DirectUpdateHandler2
> >> rollback
> >> INFO: start rollback
> >> Nov 11, 2009 3:35:44 PM org.apache.solr.update.DirectUpdateHandler2
> >> rollback
> >> INFO: end_rollback
> >>
> >>
> >
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>
Re: ${dataimporter.delta.twitter_id} not getting populated in
deltaImportQuery
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
are you sure the data comes back in the same name. Some DBs return the
field names in ALL CAPS
you may try out a delta_import using a full import too
http://wiki.apache.org/solr/DataImportHandlerFaq#My_delta-import_goes_out_of_memory_._Any_workaround_.3F
On Wed, Nov 11, 2009 at 9:55 PM, Mark Ellul <ma...@catalystic.com> wrote:
> I have 2 entities from the root node, not sure if that makes a difference!
>
> On Wed, Nov 11, 2009 at 4:49 PM, Mark Ellul <ma...@catalystic.com> wrote:
>
>> Hi,
>>
>> I have a interesting issue...
>>
>> Basically I am trying to delta imports on solr 1.4 on a postgresql 8.3
>> database.
>>
>> Basically when I am running a delta import with the entity below I get an
>> exception (see below the entity definition) showing the query its trying to
>> run and you can see that its not populating the where clause of my
>> dataImportQuery.
>>
>> I have tried ${dataimporter.delta.twitter_id} and ${dataimporter.delta.id}
>> and get the same exceptions.
>>
>> Am I missing something obvious?
>>
>> Any help would be appreciated!
>>
>> Regards
>>
>> Mark
>>
>>
>> <entity name="Tweeter" pk="twitter_id"
>> query="
>> select twitter_id,
>> twitter_id as pk,
>> 1 as site_id,
>> screen_name
>>
>> from api_tweeter WHERE
>> tweet_mapreduce_on IS NOT NULL;
>> " transformer="TemplateTransformer"
>>
>> deltaImportQuery="
>> select twitter_id,
>> twitter_id as pk,
>> 1 as site_id,
>> screen_name
>>
>> from api_tweeter
>> where twitter_id=${dataimporter.delta.twitter_id };
>> "
>> deltaQuery ="select twitter_id from api_tweeter where modified_on >
>> '${dataimporter.last_index_time}' and tweet_mapreduce_on IS NOT NULL;"
>>
>> >
>> <field name="twitter_id" column="twitter_id" />
>>
>>
>> </entity>
>>
>>
>> INFO: Completed parentDeltaQuery for Entity: Tweeter
>> Nov 11, 2009 3:35:44 PM org.apache.solr.handler.dataimport.DocBuilder
>> buildDocument
>> SEVERE: Exception while processing: Tweeter document :
>> SolrInputDocument[{}]
>> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
>> execute query: select twitter_id, twitter_id
>> as pk, 1 as site_id, screen_name from api_tweeter where
>> twitter_id=; Processing Document # 1
>> at
>> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
>> at
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
>> at
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
>> at
>> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>> at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>> Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at end of
>> input
>> Position: 1197
>> at
>> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
>> at
>> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
>> at
>> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:345)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:246)
>> ... 11 more
>> Nov 11, 2009 3:35:44 PM org.apache.solr.handler.dataimport.DataImporter
>> doDeltaImport
>> SEVERE: Delta Import Failed
>> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
>> execute query: select twitter_id, twitter_id
>> as pk, 1 as site_id, screen_name from api_tweeter where
>> twitter_id=; Processing Document # 1
>> at
>> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
>> at
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
>> at
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
>> at
>> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>> at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>> Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at end of
>> input
>> Position: 1197
>> at
>> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
>> at
>> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
>> at
>> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:345)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:246)
>> ... 11 more
>> Nov 11, 2009 3:35:44 PM org.apache.solr.update.DirectUpdateHandler2
>> rollback
>> INFO: start rollback
>> Nov 11, 2009 3:35:44 PM org.apache.solr.update.DirectUpdateHandler2
>> rollback
>> INFO: end_rollback
>>
>>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: ${dataimporter.delta.twitter_id} not getting populated in
deltaImportQuery
Posted by Mark Ellul <ma...@catalystic.com>.
I have 2 entities from the root node, not sure if that makes a difference!
On Wed, Nov 11, 2009 at 4:49 PM, Mark Ellul <ma...@catalystic.com> wrote:
> Hi,
>
> I have a interesting issue...
>
> Basically I am trying to delta imports on solr 1.4 on a postgresql 8.3
> database.
>
> Basically when I am running a delta import with the entity below I get an
> exception (see below the entity definition) showing the query its trying to
> run and you can see that its not populating the where clause of my
> dataImportQuery.
>
> I have tried ${dataimporter.delta.twitter_id} and ${dataimporter.delta.id}
> and get the same exceptions.
>
> Am I missing something obvious?
>
> Any help would be appreciated!
>
> Regards
>
> Mark
>
>
> <entity name="Tweeter" pk="twitter_id"
> query="
> select twitter_id,
> twitter_id as pk,
> 1 as site_id,
> screen_name
>
> from api_tweeter WHERE
> tweet_mapreduce_on IS NOT NULL;
> " transformer="TemplateTransformer"
>
> deltaImportQuery="
> select twitter_id,
> twitter_id as pk,
> 1 as site_id,
> screen_name
>
> from api_tweeter
> where twitter_id=${dataimporter.delta.twitter_id };
> "
> deltaQuery ="select twitter_id from api_tweeter where modified_on >
> '${dataimporter.last_index_time}' and tweet_mapreduce_on IS NOT NULL;"
>
> >
> <field name="twitter_id" column="twitter_id" />
>
>
> </entity>
>
>
> INFO: Completed parentDeltaQuery for Entity: Tweeter
> Nov 11, 2009 3:35:44 PM org.apache.solr.handler.dataimport.DocBuilder
> buildDocument
> SEVERE: Exception while processing: Tweeter document :
> SolrInputDocument[{}]
> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
> execute query: select twitter_id, twitter_id
> as pk, 1 as site_id, screen_name from api_tweeter where
> twitter_id=; Processing Document # 1
> at
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
> at
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
> at
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
> Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at end of
> input
> Position: 1197
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:345)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:246)
> ... 11 more
> Nov 11, 2009 3:35:44 PM org.apache.solr.handler.dataimport.DataImporter
> doDeltaImport
> SEVERE: Delta Import Failed
> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
> execute query: select twitter_id, twitter_id
> as pk, 1 as site_id, screen_name from api_tweeter where
> twitter_id=; Processing Document # 1
> at
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
> at
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
> at
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
> Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at end of
> input
> Position: 1197
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:345)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:246)
> ... 11 more
> Nov 11, 2009 3:35:44 PM org.apache.solr.update.DirectUpdateHandler2
> rollback
> INFO: start rollback
> Nov 11, 2009 3:35:44 PM org.apache.solr.update.DirectUpdateHandler2
> rollback
> INFO: end_rollback
>
>