You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by "Hoss Man (Confluence)" <co...@apache.org> on 2013/09/27 01:13:00 UTC
[CONF] Apache Solr Reference Guide > Uploading Structured Data
Store Data with the Data Import Handler
Space: Apache Solr Reference Guide (https://cwiki.apache.org/confluence/display/solr)
Page: Uploading Structured Data Store Data with the Data Import Handler (https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler)
Change Comment:
---------------------------------------------------------------------
don't link
Edited by Hoss Man:
---------------------------------------------------------------------
{section}
{column:width=75%}
Many search applications store the content to be indexed in a structured data store, such as a relational database. The Data Import Handler (DIH) provides a mechanism for importing content from a data store and indexing it. In addition to relational databases, DIH can index content from HTTP based data sources such as RSS and ATOM feeds, e-mail repositories, and structured XML where an XPath processor is used to generate fields.
{note}
The DataImportHandler jars are no longer included in the Solr WAR. You should add them to Solr's lib directory, or reference them via the {{<lib>}} directive in {{solrconfig.xml}}.
{note}
For more information about the Data Import Handler, see [https://wiki.apache.org/solr/DataImportHandler].
{column}
{column:width=25%}
{panel}
Topics covered in this section:
{toc:minLevel=2|maxLevel=2}
{panel}
{column}
{section}
h2. Concepts and Terminology
Descriptions of the Data Import Handler use several familiar terms, such as entity and processor, in specific ways, as explained in the table below.
|| Term || Definition ||
| Datasource | As its name suggests, a datasource defines the location of the data of interest. For a database, it's a DSN. For an HTTP datasource, it's the base URL. |
| Entity | Conceptually, an entity is processed to generate a set of documents, containing multiple fields, which (after optionally being transformed in various ways) are sent to Solr for indexing. For a RDBMS data source, an entity is a view or table, which would be processed by one or more SQL statements to generate a set of rows (documents) with one or more columns (fields). |
| Processor | An entity processor does the work of extracting content from a data source, transforming it, and adding it to the index. Custom entity processors can be written to extend or replace the ones supplied. |
| Transformer | Each set of fields fetched by the entity may optionally be transformed. This process can modify the fields, create new fields, or generate multiple rows/documents form a single row. There are several built-in transformers in the DIH, which perform functions such as modifying dates and stripping HTML. It is possible to write custom transformers using the publicly available interface. |
h2. Configuration
h3. Configuring {{solrconfig.xml}}
The Data Import Handler has to be registered in {{solrconfig.xml}}. For example:
{code:xml|borderStyle=solid|borderColor=#666666}
<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">/path/to/my/DIHconfigfile.xml</str>
</lst>
</requestHandler>
{code}
The only required parameter is the {{config}} parameter, which specifies the location of the DIH configuration file that contains specifications for the data source, how to fetch data, what data to fetch, and how to process it to generate the Solr documents to be posted to the index.
You can have multiple DIH configuration files. Each file would require a separate definition in the {{solrconfig.xml}} file, specifying a path to the file.
h3. Configuring the DIH Configuration File
There is a sample DIH application distributed with Solr in the directory {{example/example-DIH}}. This accesses a small hsqldb database. Details of how to run this example can be found in the README.txt file. The sample DIH configuration can be found in {{example/example-DIH/solr/db/conf/db-data-config.xml}}.
An annotated configuration file, based on the sample, is shown below. It extracts fields from the four tables defining a simple product database, with this schema. More information about the parameters and options shown here are described in the sections following.
{code:xml|borderStyle=solid|borderColor=#666666}
<dataConfig>
<!-- The first element is the dataSource, in this case an HSQLDB database.
The path to the JDBC driver and the JDBC URL and login credentials are all specified here.
Other permissible attributes include whether or not to autocommit to Solr,the batchsize
used in the JDBC connection, a 'readOnly' flag -->
<dataSource driver="org.hsqldb.jdbcDriver" url="jdbc:hsqldb:./example-DIH/hsqldb/ex" user="sa" />
<!-- a 'document' element follows, containing multiple 'entity' elements.
Note that 'entity' elements can be nested, and this allows the entity
relationships in the sample database to be mirrored here, so that we can
generate a denormalized Solr record which may include multiple features
for one item, for instance -->
<document>
<!-- The possible attributes for the entity element are described below.
Entity elements may contain one or more 'field' elements, which map
the data source field names to Solr fields, and optionally specify
per-field transformations -->
<!-- this entity is the 'root' entity. -->
<entity name="item" query="select * from item"
deltaQuery="select id from item where last_modified > '${dataimporter.last_index_time}'">
<field column="NAME" name="name" />
<!-- This entity is nested and reflects the one-to-many relationship between an item and its multiple features.
Note the use of variables; ${item.ID} is the value of the column 'ID' for the current item
('item' referring to the entity name) -->
<entity name="feature"
query="select DESCRIPTION from FEATURE where ITEM_ID='${item.ID}'"
deltaQuery="select ITEM_ID from FEATURE where last_modified > '${dataimporter.last_index_time}'"
parentDeltaQuery="select ID from item where ID=${feature.ITEM_ID}">
<field name="features" column="DESCRIPTION" />
</entity>
<entity name="item_category"
query="select CATEGORY_ID from item_category where ITEM_ID='${item.ID}'"
deltaQuery="select ITEM_ID, CATEGORY_ID from item_category where last_modified > '${dataimporter.last_index_time}'"
parentDeltaQuery="select ID from item where ID=${item_category.ITEM_ID}">
<entity name="category"
query="select DESCRIPTION from category where ID = '${item_category.CATEGORY_ID}'"
deltaQuery="select ID from category where last_modified > '${dataimporter.last_index_time}'"
parentDeltaQuery="select ITEM_ID, CATEGORY_ID from item_category where CATEGORY_ID=${category.ID}">
<field column="description" name="cat" />
</entity>
</entity>
</entity>
</document>
</dataConfig>
{code}
Datasources can still be specified in {{solrconfig.xml}}. These must be specified in the defaults section of the handler in {{solrconfig.xml}}. However, these are not parsed until the main configuration is loaded.
The entire configuration itself can be passed as a request parameter using the {{dataConfig}} parameter rather than using a file. When configuration errors are encountered, the error message is returned in XML format.
In Solr 4.1, a new property was added, the {{propertyWriter}} element, which allows defining the date format and locale for use with delta queries. It also allows customizing the name and location of the properties file.
The {{reload-config}} command is still supported, which is useful for validating a new configuration file, or if you want to specify a file, load it, and not have it reloaded again on import. If there is an {{xml}} mistake in the configuration a user-friendly message is returned in {{xml}} format. You can then fix the problem and do a {{reload-config}}.
{tip}
You can also view the DIH configuration in the Solr Admin UI. There is also an interface to import content.
{tip}
h2. Data Import Handler Commands
DIH commands are sent to Solr via an HTTP request. The following operations are supported.
|| Command || Description ||
| {{abort}} | Aborts an ongoing operation. The URL is {{http://<host>:<port>/solr/dataimport?command=abort}}. |
| delta-import | For incremental imports and change detection. The command is of the form {{http://<host>:<port>/solr/dataimport?command=delta-import}}. It supports the same clean, commit, optimize and debug parameters as full-import command. |
| {{full-import}} | A Full Import operation can be started with a URL of the form {{http://<host>:<port>/solr/dataimport?command=full-import}}. The command returns immediately. The operation will be started in a new thread and the _status_ attribute in the response should be shown as _busy_. The operation may take some time depending on the size of dataset. Queries to Solr are not blocked during full-imports. \\
When a full-import command is executed, it stores the start time of the operation in a file located at {{conf/dataimport.properties}}. This stored timestamp is used when a delta-import operation is executed. \\
For a list of parameters that can be passed to this command, see below. |
| {{reload-config}} | If the configuration file has been changed and you wish to reload it without restarting Solr, run the command {{http://<host>:<port>/solr/dataimport?command=reload-config}}. |
| {{status}} | The URL is {{http://<host>:<port>/solr/dataimport?command=status}}. It returns statistics on the number of documents created, deleted, queries run, rows fetched, status, and so on. |
h3. Parameters for the {{full-import}} Command
The {{full-import}} command accepts the following parameters:
|| Parameter || Description ||
| clean | Default is true. Tells whether to clean up the index before the indexing is started. |
| commit | Default is true. Tells whether to commit after the operation. |
| debug | Default is false Runs the command in debug mode. It is used by the interactive development mode. Note that in debug mode, documents are never committed automatically. If you want to run debug mode and commit the results too, add {{commit=true}} as a request parameter. |
| entity | The name of an entity directly under the {{<document>}} tag in the configuration file. Use this to execute one or more entities selectively. Multiple "entity" parameters can be passed on to run multiple entities at once. If nothing is passed, all entities are executed. |
| optimize | Default is true. Tells Solr whether to optimize after the operation. |
h2. Property Writer
The {{propertyWriter}} element defines the date format and locale for use with delta queries. It is an optional configuration. Add the element to the DIH configuration file, directly under the {{dataConfig}} element.
{code:xml|borderStyle=solid|borderColor=#666666}
<propertyWriter dateFormat="yyyy-MM-dd HH:mm:ss" type="SimplePropertiesWriter" directory="data" filename="my_dih.properties" locale="en_US" />
{code}
The parameters available are:
|| Parameter || Description ||
| dateFormat | A java.text.SimpleDateFormat to use when converting the date to text. The default is "yyyy-MM-dd HH:mm:ss". |
| type | The implementation class. Use {{SimplePropertiesWriter}} for non-SolrCloud installations. If using SolrCloud, use {{ZKPropertiesWriter}}. If this is not specified, it will default to the appropriate class depending on if SolrCloud mode is enabled. |
| directory | Used with the {{SimplePropertiesWriter}} only). The directory for the properties file. If not specified, the default is "conf". |
| filename | Used with the {{SimplePropertiesWriter}} only). The name of the properties file. If not specified, the default is the requestHandler name (as defined in {{solrconfig.xml}}, appended by ".properties" (i.e., "dataimport.properties").|
| locale | The locale. If not defined, the ROOT locale is used. It must be specified as language-country. For example, {{en-US}}. |
h2. Data Sources
A data source specifies the origin of data and its type. Somewhat confusingly, some data sources are configured within the associated entity processor. Data sources can also be specified in {{solrconfig.xml}}, which is useful when you have multiple environments (for example, development, QA, and production) differing only in their data sources.
You can create a custom data source by writing a class that extends {{org.apache.solr.handler.dataimport.DataSource}}.
The mandatory attributes for a data source definition are its name and type. The name identifies the data source to an Entity element.
The types of data sources available are described below.
h3. ContentStreamDataSource
This takes the POST data as the data source. This can be used with any EntityProcessor that uses a {{DataSource<Reader>}}.
h3. FieldReaderDataSource
This can be used where a database field contains XML which you wish to process using the XpathEntityProcessor. You would set up a configuration with both JDBC and FieldReader data sources, and two entities, as follows:
{code:xml|borderStyle=solid|borderColor=#666666}
<dataSource name="a1" driver="org.hsqldb.jdbcDriver" ... />
<dataSource name="a2" type=FieldReaderDataSource" />
<!-- processor for database -->
<entity name ="e1" dataSource="a1" processor="SQLEntityProcessor" pk="docid"
query="select * from t1 ...">
<!-- nested XpathEntity; the field in the parent which is to be used for
Xpath is set in the "datafield" attribute in place of the "url" attribute -->
<entity name="e2"
dataSource="a2"
processor="XPathEntityProcessor"
dataField="e1.fieldToUseForXPath"
<!-- Xpath configuration follows -->
...
</entity>
</entity>
{code}
The FieldReaderDataSource can take an {{encoding}} parameter, which will default to "UTF-8" if not specified.It must be specified as language-country. For example, {{en-US}}.
h3. FileDataSource
This can be used like an [URLDataSource|#URLDataSource], but is used to fetch content from files on disk. The only difference from URLDataSource, when accessing disk files, is how a pathname is specified.
This data source accepts these optional attributes.
|| Optional Attribute || Description ||
| basePath | The base path relative to which the value is evaluated if it is not absolute. |
| encoding | Defines the character encoding to use. If not defined, UTF-8 is used. |
h3. JdbcDataSource
This is the default datasource. It's used with the [SQLEntityProcessor|#The SQL Entity Processor]. See the example in the [FieldReaderDataSource|#FieldReaderDataSource] section for details on configuration.
h3. URLDataSource
This data source is often used with XPathEntityProcessor to fetch content from an underlying {{file}}{{:}}{{//}} or {{http://}} location. Here's an example:
{code:xml|borderStyle=solid|borderColor=#666666}
<dataSource name="a"
type="URLDataSource"
baseUrl="http://host:port/"
encoding="UTF-8"
connectionTimeout="5000"
readTimeout="10000"/>
{code}
The URLDataSource type accepts these optional parameters:
|| Optional Parameter || Description ||
| baseURL | Specifies a new baseURL for pathnames. You can use this to specify host/port changes between Dev/QA/Prod environments. Using this attribute isolates the changes to be made to the {{solrconfig.xml}} |
| connectionTimeout | Specifies the length of time in milliseconds after which the connection should time out. The default value is 5000ms. |
| encoding | By default the encoding in the response header is used. You can use this property to override the default encoding. |
| readTimeout | Specifies the length of time in milliseconds after which a read operation should time out. The default value is 10000ms. |
h2. Entity Processors
Entity processors extract data, transform it, and add it to a Solr index. Examples of entities include views or tables in a data store.
Each processor has its own set of attributes, described in its own section below. In addition, there are non-specific attributes common to all entities which may be specified.
|| Attribute || Use ||
| datasource | The name of a data source. Used if there are multiple data sources, specified, in which case each one must have a name. |
| name | Required. The unique name used to identify an entity. |
| pk | The primary key for the entity. It is optional, and required only when using delta-imports. It has no relation to the uniqueKey defined in {{schema.xml}} but they can both be the same. It is mandatory if you do delta-imports and then refers to the column name in {{$\{dataimporter.delta.<column-name>\}}} which is used as the primary key. |
| processor | Default is SQLEntityProcessor. Required only if the datasource is not RDBMS. |
| onError | Permissible values are (abort\|skip\|continue) . The default value is 'abort'. 'Skip' skips the current document. 'Continue' ignores the error and processing continues. |
| preImportDeleteQuery | Before a full-import command, use this query this to cleanup the index instead of using '*:*'. This is honored only on an entity that is an immediate sub-child of {{<document>}}. |
| postImportDeleteQuery | Similar to the above, but executed after the import has completed. |
| rootEntity | By default the entities immediately under the {{<document>}} are root entities. If this attribute is set to false, the entity directly falling under that entity will be treated as the root entity (and so on). For every row returned by the root entity, a document is created in Solr. |
| transformer | Optional. One or more transformers to be applied on this entity. |
h3. The SQL Entity Processor
The SqlEntityProcessor is the default processor. The associated [data source|#JdbcDataSource] should be a JDBC URL.
The entity attributes specific to this processor are shown in the table below.
|| Attribute || Use ||
| query | Required. The SQL query used to select rows. |
| deltaQuery | SQL query used if the operation is delta-import. This query selects the primary keys of the rows which will be parts of the delta-update. The pks will be available to the deltaImportQuery through the variable {{$\{dataimporter.delta.<column-name>\}}}. |
| parentDeltaQuery | SQL query used if the operation is delta-import. |
| deletedPkQuery | SQL query used if the operation is delta-import. |
| deltaImportQuery | SQL query used if the operation is delta-import. If this is not present, DIH tries to construct the import query by(after identifying the delta) modifying the 'query' (this is error prone). There is a namespace {{$\{dataimporter.delta.<column-name>\}}} which can be used in this query. For example, {{select * from tbl where id=$\{dataimporter.delta.id\}}}. |
h3. The XPathEntityProcessor
This processor is used when indexing XML formatted data. The data source is typically [URLDataSource|#URLDataSource] or [FileDataSource|#FileDataSource]. Xpath can also be used with the [FileListEntityProcessor|#The FileListEntityProcessor] described below, to generate a document from each file.
The entity attributes unique to this processor are shown below.
|| Attribute || Use ||
| Processor | Required. Must be set to "XpathEntityProcessor". |
| url | Required. HTTP URL or file location. |
| stream | Optional: Set to true for a large file or download. |
| forEach | Required unless you define {{useSolrAddSchema}}. The Xpath expression which demarcates each record. This will be used to set up the processing loop. |
| xsl | Optional: Its value (a URL or filesystem path) is the name of a resource used as a preprocessor for applying the XSL transformation. |
| useSolrAddSchema | Set this to true if the content is in the form of the standard Solr update XML schema. |
| flatten | Optional: If set true, then text from under all the tags is extracted into one field. |
Each field element in the entity can have the following attributes as well as the default ones.
|| Attribute || Use ||
| xpath | Required. The XPath expression which will extract the content from the record for this field. Only a subset of Xpath syntax is supported. |
| commonField | Optional. If true, then when this field is encountered in a record it will be copied to future records when creating a Solr document. |
Example:
{code:xml|borderStyle=solid|borderColor=#666666}
<!-- slashdot RSS Feed --->
<dataConfig>
<dataSource type="HttpDataSource" />
<document>
<entity name="slashdot"
pk="link"
url="http://rss.slashdot.org/Slashdot/slashdot"
processor="XPathEntityProcessor"
<!-- forEach sets up a processing loop ; here there are two expressions-->
forEach="/RDF/channel | /RDF/item"
transformer="DateFormatTransformer">
<field column="source"
xpath="/RDF/channel/title"
commonField="true" />
<field column="source-link"
xpath="/RDF/channel/link"
commonField="true"/>
<field column="subject"
xpath="/RDF/channel/subject"
commonField="true" />
<field column="title"
xpath="/RDF/item/title" />
<field column="link"
xpath="/RDF/item/link" />
<field column="description"
xpath="/RDF/item/description" />
<field column="creator"
xpath="/RDF/item/creator" />
<field column="item-subject"
xpath="/RDF/item/subject" />
<field column="date"
xpath="/RDF/item/date"
dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss" />
<field column="slash-department"
xpath="/RDF/item/department" />
<field column="slash-section"
xpath="/RDF/item/section" />
<field column="slash-comments"
xpath="/RDF/item/comments" />
</entity>
</document>
</dataConfig>
{code}
[http://wiki.apache.org/solr/MailEntityProcessor]
h3. The TikaEntityProcessor
The TikaEntityProcessor uses Apache Tika to process incoming documents. This is similar to [solr:Uploading Data with Solr Cell using Apache Tika], but using the DataImportHandler options instead.
The {{example-DIH}} directory in Solr's {{example}} directory shows one option for using the TikaEntityProcessor. Here is the sample {{data-config.xml}} file:
{code:xml|borderStyle=solid|borderColor=#666666}
<dataConfig>
<dataSource type="BinFileDataSource" />
<document>
<entity name="tika-test" processor="TikaEntityProcessor"
url="../contrib/extraction/src/test-files/extraction/solr-word.pdf" format="text">
<field column="Author" name="author" meta="true"/>
<field column="title" name="title" meta="true"/>
<field column="text" name="text"/>
</entity>
</document>
</dataConfig>
{code}
The parameters for this processor are described in the table below:
|| Attribute || Use ||
| dataSource | This parameter defines the data source and an optional name which can be referred to in later parts of the configuration if needed. This is the same dataSource explained in the description of general entity processor attributes above. \\
\\
The available data source types for this processor are:
* BinURLDataSource: used for HTTP resources, but can also be used for files.
* BinContentStreamDataSource: used for uploading content as a stream.
* BinFileDataSource: used for content on the local filesystem. |
| url | The path to the source file(s), as a file path or a traditional internet URL. This parameter is required. |
| htmlMapper | Allows control of how Tika parses HTML. The "default" mapper strips much of the HTML from documents while the "identity" mapper passes all HTML as-is with no modifications. If this parameter is defined, it must be either *default* or *identity*; if it is absent, "default" is assumed. |
| format | The output format. The options are *text*, *xml*, *html* or *none*. The default is "text" if not defined. The format "none" can be used if metadata only should be indexed and not the body of the documents. |
| parser | The default parser is {{org.apache.tika.parser.AutoDetectParser}}. If a custom or other parser should be used, it should be entered as a fully-qualified name of the class and path. |
| fields | The list of fields from the input documents and how they should be mapped to Solr fields. If the attribute {{meta}} is defined as "true", the field will be obtained from the metadata of the document and not parsed from the body of the main text. |
h3. The FileListEntityProcessor
This processor is basically a wrapper, and is designed to generate a set of files satisfying conditions specified in the attributes which can then be passed to another processor, such as the [XPathEntityProcessor|#The XPathEntityProcessor]. The entity information for this processor would be nested within the FileListEnitity entry. It generates four implicit fields: {{fileAbsolutePath}}, {{fileSize}}, {{fileLastModified}}, {{fileName}} which can be used in the nested processor. This processor does not use a data source.
The attributes specific to this processor are described in the table below:
|| Attribute || Use ||
| fileName | Required. A regular expression pattern to identify files to be included. |
| basedir | Required. The base directory (absolute path). |
| recursive | Whether to search directories recursively. Default is 'false'. |
| excludes | A regular expression pattern to identify files which will be excluded. |
| newerThan | A date in the format {{yyyy-MM-ddHH:mm:ss}} or a date math expression ({{NOW - 2YEARS}}). |
| olderThan | A date, using the same formats as newerThan. |
| rootEntity | This should be set to false. This ensures that each row (filepath) emitted by this processor is considered to be a document. |
| dataSource | Must be set to null. |
The example below shows the combination of the FileListEntityProcessor with another processor which will generate a set of fields from each file found.
{code:xml|borderStyle=solid|borderColor=#666666}
<dataConfig>
<dataSource type="FileDataSource"/><document>
<!-- this outer processor generates a list of files satisfying the conditions
specified in the attributes -->
<entity name="f" processor="FileListEntityProcessor"
fileName=".*xml"
newerThan="'NOW-30DAYS'"
recursive="true"
rootEntity="false"
dataSource="null"
baseDir="/my/document/directory">
<!-- this processor extracts content using Xpath from each file found -->
<entity name="nested" processor="XPathEntityProcessor"
forEach="/rootelement" url="${f.fileAbsolutePath}" >
<field column="name" xpath="/rootelement/name"/>
<field column="number" xpath="/rootelement/number"/>
</entity>
</entity>
</document>
</dataConfig>
{code}
h3. LineEntityProcessor
This EntityProcessor reads all content from the data source on a line by line basis and returns a field called {{rawLine}} for each line read. The content is not parsed in any way; however, you may add transformers to manipulate the data within the {{rawLine}} field, or to create other additional fields.
The lines read can be filtered by two regular expressions specified with the {{acceptLineRegex}} and {{omitLineRegex}} attributes. The table below describes the LineEntityProcessor's attributes:
|| Attribute || Description ||
| url | A required attribute that specifies the location of the input file in a way that is compatible with the configured data source. If this value is relative and you are using FileDataSource or URLDataSource, it assumed to be relative to baseLoc. |
| acceptLineRegex | An optional attribute that if present discards any line which does not match the regExp. |
| omitLineRegex | An optional attribute that is applied after any acceptLineRegex and that discards any line which matches this regExp. |
For example:
{code:xml|borderStyle=solid|borderColor=#666666}
<entity name="jc"
processor="LineEntityProcessor"
acceptLineRegex="^.*\.xml$"
omitLineRegex="/obsolete"
url="file:///Volumes/ts/files.lis"
rootEntity="false"
dataSource="myURIreader1"
transformer="RegexTransformer,DateFormatTransformer"
>
...
{code}
While there are use cases where you might need to create a Solr document for each line read from a file, it is expected that in most cases that the lines read by this processor will consist of a pathname, which in turn will be consumed by another EntityProcessor, such as [XPathEntityProcessor|http://wiki.apache.org/solr/PathEntityProcessor].
h3. PlainTextEntityProcessor
This EntityProcessor reads all content from the data source into an single implicit field called {{plainText}}. The content is not parsed in any way, however you may add transformers to manipulate the data within the {{plainText}} as needed, or to create other additional fields.
For example:
{code:xml|borderStyle=solid|borderColor=#666666}
<entity processor="PlainTextEntityProcessor" name="x" url="http://abc.com/a.txt" dataSource="data-source-name">
<!-- copies the text to a field called 'text' in Solr-->
<field column="plainText" name="text"/>
</entity>
{code}
Ensure that the dataSource is of type {{DataSource<Reader>}} ({{FileDataSource}}, {{URLDataSource}}).
h2. Transformers
Transformers manipulate the fields in a document returned by an entity. A transformer can create new fields or modify existing ones. You must tell the entity which transformers your import operation will be using, by adding an attribute containing a comma separated list to the {{<entity>}} element.
{code:xml|borderStyle=solid|borderColor=#666666}
<entity name="abcde"
transformer="org.apache.solr....,my.own.transformer,..." />
{code}
Specific transformation rules are then added to the attributes of a {{<field>}} element, as shown in the examples below. The transformers are applied in the order in which they are specified in the transformer attribute.
The Data Import Handler contains several built-in transformers. You can also write your own custom transformers, as described in the Solr Wiki (see [http://wiki.apache.org/solr/DIHCustomTransformer]). The ScriptTransformer (described below) offers an alternative method for writing your own transformers.
Solr includes the following built-in transformers:
|| Transformer Name || Use ||
| [ClobTransformer|#ClobTransformer] | Used to create a String out of a Clob type in database. |
| [DateFormatTransformer|#The DateFormatTransformer] | Parse date/time instances. |
| [HTMLStripTransformer|#The HTMLStripTransformer] | Strip HTML from a field. |
| [LogTransformer|#The LogTransformer] | Used to log data to log files or a console. |
| [NumberFormatTransformer|#The NumberFormatTransformer] | Uses the NumberFormat class in java to parse a string into a number. |
| [RegexTransformer|#The RegexTransformer] | Use regular expressions to manipulate fields. |
| [ScriptTransformer|#The ScriptTransformer] | Write transformers in Javascript or any other scripting language supported by Java. Requires Java 6. |
| [TemplateTransformer|#The TemplateTransformer] | Transform a field using a template. |
These transformers are described below.
h3. ClobTransformer
You can use the ClobTransformer to create a string out of a CLOB in a database. A CLOB is a character large object: a collection of character data typically stored in a separate location that is referenced in the database. See [http://en.wikipedia.org/wiki/Character_large_object]. Here's an example of invoking the ClobTransformer.
{code:xml|borderStyle=solid|borderColor=#666666}
<entity name="e" transformer="ClobTransformer" ..>
<field column="hugeTextField" clob="true" />
...
</entity>
{code}
The ClobTransformer accepts these attributes:
|| Attribute || Description ||
| clob | Boolean value to signal if ClobTransformer should process this field or not. If this attribute is omitted, then the corresponding field is not transformed. |
| sourceColName | The source column to be used as input. If this is absent source and target are same |
h3. The DateFormatTransformer
This transformer converts dates from one format to another. This would be useful, for example, in a situation where you wanted to convert a field with a fully specified date/time into a less precise date format, for use in faceting.
DateFormatTransformer applies only on the fields with an attribute {{dateTimeFormat}}. Other fields are not modified.
This transformer recognizes the following attributes:
|| Attribute || Description ||
| dateTimeFormat | The format used for parsing this field. This must comply with the syntax of the [JavaSimpleDateFormat|http://java.sun.com/j2se/1.4.2/docs/api/java/text/SimpleDateFormat.html] class. |
| sourceColName | The column on which the dateFormat is to be applied. If this is absent source and target are same. |
| locale | The locale to use for date transformations. If not specified, the ROOT locale will be used. It must be specified as language-country. For example, {{en-US}}. |
Here is example code that returns the date rounded up to the month "2007-JUL":
{code:xml|borderStyle=solid|borderColor=#666666}
<entity name="en" pk="id" transformer="DateTimeTransformer" ... >
...
<field column="date"
sourceColName="fulldate"
dateTimeFormat="yyyy-MMM"/>
</entity>
{code}
h3. The HTMLStripTransformer
You can use this transformer to strip HTML out of a field. For example:
{code:xml|borderStyle=solid|borderColor=#666666}
<entity name="e" transformer="HTMLStripTransformer" ..>
<field column="htmlText" stripHTML="true" />
...
</entity>
{code}
There is one attribute for this transformer, {{stripHTML}}, which is a boolean value (true/false) to signal if the HTMLStripTransformer should process the field or not.
h3. The LogTransformer
You can use this transformer to log data to the console or log files. For example:
{code:xml|borderStyle=solid|borderColor=#666666}
<entity ...
transformer="LogTransformer"
logTemplate="The name is $\{e.name\}" logLevel="debug" >
....
</entity>
{code}
Unlike other transformers, the LogTransformer does not apply to any field, so the attributes are applied on the entity itself.
h3. The NumberFormatTransformer
Use this transformer to parse a number from a string, converting it into the specified format, and optionally using a different locale.
NumberFormatTransformer will be applied only to fields with an attribute {{formatStyle}}.
This transformer recognizes the following attributes:
|| Attribute || Description ||
| formatStyle | The format used for parsing this field. The value of the attribute must be one of ({{number\|percent\|integer\|currency}}). This uses the semantics of the Java NumberFormat class. |
| sourceColName | The column on which the NumberFormat is to be applied. This is attribute is absent. The source column and the target column are the same. |
| locale | The locale to be used for parsing the strings. If this is absent, the ROOT locale is used. It must be specified as language-country. For example, {{en-US}}. |
For example:
{code:xml|borderStyle=solid|borderColor=#666666}
<entity name="en" pk="id" transformer="NumberFormatTransformer" ...>
...
<!-- treat this field as UK pounds -->
<field name="price_uk"
column="price"
formatStyle="currency"
locale="en-UK" />
</entity>
{code}
h3. The RegexTransformer
The regex transformer helps in extracting or manipulating values from fields (from the source) using Regular Expressions. The actual class name is {{org.apache.solr.handler.dataimport.RegexTransformer}}. But as it belongs to the default package the package-name can be omitted.
The table below describes the attributes recognized by the regex transformer.
|| Attribute || Description ||
| regex | The regular expression that is used to match against the column or sourceColName's value(s). If replaceWith is absent, each regex _group_ is taken as a value and a list of values is returned. |
| sourceColName | The column on which the regex is to be applied. If not present, then the source and target are identical. |
| splitBy | Used to split a string. It returns a list of values. |
| groupNames | A comma separated list of field column names, used where the regex contains groups and each group is to be saved to a different field. If some groups are not to be named leave a space between commas. |
| replaceWith | Used along with regex . It is equivalent to the method {{new String(<sourceColVal>).replaceAll(<regex>, <replaceWith>)}}. |
Here is an example of configuring the regex transformer:
{code:xml|borderStyle=solid|borderColor=#666666}
<entity name="foo" transformer="RegexTransformer"
query="select full_name , emailids from foo"/>
... />
<field column="full_name"/>
<field column="firstName" regex="Mr(\w*)\b.*" sourceColName="full_name"/>
<field column="lastName" regex="Mr.*?\b(\w*)" sourceColName="full_name"/>
<!-- another way of doing the same -->
<field column="fullName" regex="Mr(\w*)\b(.*)" groupNames="firstName,lastName"/>
<field column="mailId" splitBy="," sourceColName="emailids"/>
</entity>
{code}
In this example, regex and sourceColName are custom attributes used by the transformer. The transformer reads the field {{full_name}} from the resultset and transforms it to two new target fields, {{firstName}} and {{lastName}}. Even though the query returned only one column, {{full_name}}, in the result set, the Solr document gets two extra fields {{firstName}} and {{lastName}} which are "derived" fields. These new fields are only created if the regexp matches.
The emailids field in the table can be a comma-separated value. It ends up producing one or more email IDs, and we expect the {{mailId}} to be a multivalued field in Solr.
Note that this transformer can either be used to split a string into tokens based on a splitBy pattern, or to perform a string substitution as per replaceWith, or it can assign groups within a pattern to a list of groupNames. It decides what it is to do based upon the above attributes {{splitBy}}, {{replaceWith}} and {{groupNames}} which are looked for in order. This first one found is acted upon and other unrelated attributes are ignored.
h3. The ScriptTransformer
The script transformer allows arbitrary transformer functions to be written in any scripting language supported by Java, such as Javascript, JRuby, Jython, Groovy, or BeanShell. Javascript is integrated into Java 6; you'll need to integrate other languages yourself.
Each function you write must accept a row variable (which corresponds to a {{Java Map<String,Object>}}, thus permitting {{get,put,remove}} operations). Thus you can modify the value of an existing field or add new fields. The return value of the function is the returned object.
The script is inserted into the DIH configuration file at the top level and is called once for each row.
Here is a simple example.
{code:xml|borderStyle=solid|borderColor=#666666}
<dataconfig>
<!-- simple script to generate a new row, converting a temperature from Fahrenheit to Centigrade -->
<script>
<CDATA
function f2c(row) { var tempf, tempc; tempf = row.get('temp_f'); if (tempf != null) { tempc = (tempf - 32.0)*5.0/9.0
row.put('temp_c', temp_c);
}
return row;
}
>
</script>
<document>
<!-- the function is specified as an entity attribute -->
<entity name="e1" pk="id" transformer="script:f2c" query="select * from X">
....
</entity>
</document>
</dataConfig>
{code}
h3. The TemplateTransformer
You can use the template transformer to construct or modify a field value, perhaps using the value of other fields. You can insert extra text into the template.
{code:xml|borderStyle=solid|borderColor=#666666}
<entity name="en" pk="id" transformer="TemplateTransformer" ...>
...
<!-- generate a full address from fields containing the component parts -->
<field column="full_address"
template="$en.\{street\},$en\{city\},$en\{zip\}" />
</entity>
{code}
h2. Special Commands for the Data Import Handler
You can pass special commands to the DIH by adding any of the variables listed below to any row returned by any component:
|| Variable || Description ||
| $skipDoc | Skip the current document; that is, do not add it to Solr. The value can be the string {{true\|false}}. |
| $skipRow | Skip the current row. The document will be added with rows from other entities. The value can be the string {{true\|false}} |
| $docBoost | Boost the current document. The boost value can be a number or the {{toString}} conversion of a number. |
| $deleteDocById | Delete a document from Solr with this ID. The value has to be the {{uniqueKey}} value of the document. |
| $deleteDocByQuery | Delete documents from Solr using this query. The value must be a Solr Query. |
{scrollbar}
Stop watching space: https://cwiki.apache.org/confluence/users/removespacenotification.action?spaceKey=solr
Change email notification preferences: https://cwiki.apache.org/confluence/users/editmyemailsettings.action