You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Julio Castillo <jc...@edgenuity.com> on 2008/06/11 17:54:54 UTC
RE: [jira] Updated: (SOLR-469) Data Import RequestHandler
Shalin,
Thanks for consolidating the patch.
Any idea, when the dB Import request handler will be part of the nightly
build?
Thanks again
** julio
-----Original Message-----
From: Shalin Shekhar Mangar (JIRA) [mailto:jira@apache.org]
Sent: Wednesday, June 11, 2008 8:43 AM
To: jcastillo@edgenuity.com
Subject: [jira] Updated: (SOLR-469) Data Import RequestHandler
[
https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugi
n.system.issuetabpanels:all-tabpanel ]
Shalin Shekhar Mangar updated SOLR-469:
---------------------------------------
Attachment: SOLR-469.patch
A new patch file (SOLR-469.patch) consisting of some important bug fixes and
minor enhancements. The changes and the corresponding classes are given
below
*Changes*
* Set fetch size to Integer.MIN_VALUE if batchSize in configuration is -1 as
per Patrick's suggestion -- JdbcDataSource
* Transformers can add a boost to a document by adding a key/value pair
row.put("$docBoost", 2.0f) from any entity -- DocBuilder,SolrWriter and
DataImportHandler
* Fixes for infinite loop in SqlEntityProcessor when delta query fails for
some reason and NullPointerException is thrown in EntityProcessorBase --
EntityProcessorBase
* Fix for NullPointerException in TemplateTransformer and corresponding test
-- TemplateTransformer, TestTemplateTransformer
* Enhancement for specifying table.column syntax for pk attribute in entity
as per issue reported by Chris Moser and Olivier Poitrey --
SqlEntityProcessor,TestSqlEntityProcessor2
* Fix for NullPointerException in XPathRecordReader when attribute specified
through xpath is null -- XPathRecordReader, TestXPathRecordReader
* Enhancement to DataSource interface to provide a close method --
DataSource, FileDataSource, HttpDataSource, MockDataSource
* Context interface has a new method getDataSource(String entityName) for
getting a new DataSource instance for the given entity -- Context,
ContextImpl, DataImporter, DocBuilder
* FileListEntityProcessor implements olderThan and newerThan filtering
parameters -- FileListEntityProcessor, TestFileListEntityProcessor
* Debug Mode can be disabled from solrconfig.xml by enableDebug=false --
DataImporter, DataImportHandler
* Running statistics are exposed on the Solr Statistics page in addition to
cumulative statictics -- DataImportHandler, DocBuilder
> Data Import RequestHandler
> --------------------------
>
> Key: SOLR-469
> URL: https://issues.apache.org/jira/browse/SOLR-469
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 1.3
> Reporter: Noble Paul
> Assignee: Grant Ingersoll
> Fix For: 1.3
>
> Attachments: SOLR-469-contrib.patch, SOLR-469.patch,
> SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch,
> SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch
>
>
> We need a RequestHandler Which can import data from a DB or other
dataSources into the Solr index .Think of it as an advanced form of
SqlUpload Plugin (SOLR-103).
> The way it works is as follows.
> * Provide a configuration file (xml) to the Handler which takes in the
necessary SQL queries and mappings to a solr schema
> - It also takes in a properties file for the data source
configuraution
> * Given the configuration it can also generate the solr schema.xml
> * It is registered as a RequestHandler which can take two commands
do-full-import, do-delta-import
> - do-full-import - dumps all the data from the Database into
the index (based on the SQL query in configuration)
> - do-delta-import - dumps all the data that has changed since
last import. (We assume a modified-timestamp column in tables)
> * It provides a admin page
> - where we can schedule it to be run automatically at regular
intervals
> - It shows the status of the Handler (idle, full-import,
> delta-import)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Updated: (SOLR-469) Data Import RequestHandler
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Hi Julio,
That was fast! I just uploaded a patch :)
Actually, it is waiting on SOLR-563 (
http://issues.apache.org/jira/browse/SOLR-563) which deals with modifying
the build scripts to create a contrib project area in Solr. I'm planning to
work on that this week. Once that is done, it would be upto a committer to
add it to the trunk.
On Wed, Jun 11, 2008 at 9:24 PM, Julio Castillo <jc...@edgenuity.com>
wrote:
> Shalin,
> Thanks for consolidating the patch.
>
> Any idea, when the dB Import request handler will be part of the nightly
> build?
>
> Thanks again
>
> ** julio
>
> -----Original Message-----
> From: Shalin Shekhar Mangar (JIRA) [mailto:jira@apache.org]
> Sent: Wednesday, June 11, 2008 8:43 AM
> To: jcastillo@edgenuity.com
> Subject: [jira] Updated: (SOLR-469) Data Import RequestHandler
>
>
> [
>
> https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugi
> n.system.issuetabpanels:all-tabpanel ]
>
> Shalin Shekhar Mangar updated SOLR-469:
> ---------------------------------------
>
> Attachment: SOLR-469.patch
>
> A new patch file (SOLR-469.patch) consisting of some important bug fixes
> and
> minor enhancements. The changes and the corresponding classes are given
> below
>
> *Changes*
> * Set fetch size to Integer.MIN_VALUE if batchSize in configuration is -1
> as
> per Patrick's suggestion -- JdbcDataSource
> * Transformers can add a boost to a document by adding a key/value pair
> row.put("$docBoost", 2.0f) from any entity -- DocBuilder,SolrWriter and
> DataImportHandler
> * Fixes for infinite loop in SqlEntityProcessor when delta query fails for
> some reason and NullPointerException is thrown in EntityProcessorBase --
> EntityProcessorBase
> * Fix for NullPointerException in TemplateTransformer and corresponding
> test
> -- TemplateTransformer, TestTemplateTransformer
> * Enhancement for specifying table.column syntax for pk attribute in entity
> as per issue reported by Chris Moser and Olivier Poitrey --
> SqlEntityProcessor,TestSqlEntityProcessor2
> * Fix for NullPointerException in XPathRecordReader when attribute
> specified
> through xpath is null -- XPathRecordReader, TestXPathRecordReader
> * Enhancement to DataSource interface to provide a close method --
> DataSource, FileDataSource, HttpDataSource, MockDataSource
> * Context interface has a new method getDataSource(String entityName) for
> getting a new DataSource instance for the given entity -- Context,
> ContextImpl, DataImporter, DocBuilder
> * FileListEntityProcessor implements olderThan and newerThan filtering
> parameters -- FileListEntityProcessor, TestFileListEntityProcessor
> * Debug Mode can be disabled from solrconfig.xml by enableDebug=false --
> DataImporter, DataImportHandler
> * Running statistics are exposed on the Solr Statistics page in addition to
> cumulative statictics -- DataImportHandler, DocBuilder
>
> > Data Import RequestHandler
> > --------------------------
> >
> > Key: SOLR-469
> > URL: https://issues.apache.org/jira/browse/SOLR-469
> > Project: Solr
> > Issue Type: New Feature
> > Components: update
> > Affects Versions: 1.3
> > Reporter: Noble Paul
> > Assignee: Grant Ingersoll
> > Fix For: 1.3
> >
> > Attachments: SOLR-469-contrib.patch, SOLR-469.patch,
> > SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch,
> > SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch
> >
> >
> > We need a RequestHandler Which can import data from a DB or other
> dataSources into the Solr index .Think of it as an advanced form of
> SqlUpload Plugin (SOLR-103).
> > The way it works is as follows.
> > * Provide a configuration file (xml) to the Handler which takes in
> the
> necessary SQL queries and mappings to a solr schema
> > - It also takes in a properties file for the data source
> configuraution
> > * Given the configuration it can also generate the solr schema.xml
> > * It is registered as a RequestHandler which can take two commands
> do-full-import, do-delta-import
> > - do-full-import - dumps all the data from the Database into
> the index (based on the SQL query in configuration)
> > - do-delta-import - dumps all the data that has changed since
> last import. (We assume a modified-timestamp column in tables)
> > * It provides a admin page
> > - where we can schedule it to be run automatically at regular
> intervals
> > - It shows the status of the Handler (idle, full-import,
> > delta-import)
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
--
Regards,
Shalin Shekhar Mangar.