You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Sascha Szott (JIRA)" <ji...@apache.org> on 2009/08/11 19:28:14 UTC
[jira] Created: (SOLR-1358) Integration of Solr Cell and
DataImportHandler
Integration of Solr Cell and DataImportHandler
----------------------------------------------
Key: SOLR-1358
URL: https://issues.apache.org/jira/browse/SOLR-1358
Project: Solr
Issue Type: New Feature
Components: contrib - DataImportHandler
Reporter: Sascha Szott
At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Akshay K. Ukey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Akshay K. Ukey updated SOLR-1358:
---------------------------------
Attachment: SOLR-1358.patch
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
> Attachments: SOLR-1358.patch, SOLR-1358.patch, SOLR-1358.patch
>
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noble Paul updated SOLR-1358:
-----------------------------
Attachment: SOLR-1358.patch
cleaned a bit
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
> Attachments: SOLR-1358.patch, SOLR-1358.patch
>
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noble Paul updated SOLR-1358:
-----------------------------
Attachment: SOLR-1358.patch
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
> Attachments: SOLR-1358.patch, SOLR-1358.patch
>
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750855#action_12750855 ]
Noble Paul edited comment on SOLR-1358 at 12/9/09 4:48 AM:
-----------------------------------------------------------
Let us provide a new TikaEntityProcessor
{code:xml}
<dataConfig>
<!-- use any of type DataSource<InputStream> -->
<dataSource type="BinURLDataSource"/>
<document>
<!-- The value of format can be text|xml|html|none. this is the format in which the body is emited (the 'text' field) . The implicit field 'text' will have that format.
default value is 'text' (if not specified) . format="none" means body is not emited-->
<entity processor="TikaEntityProcessor" tikaConfig="tikaconfig.xml" url="${some.var.goes.here}" format="text">
<!--Do appropriate mapping here meta="true" means it is a metadata field -->
<field column="Author" meta="true" name="author"/>
<field column="title" meta="true" name="docTitle"/>
<!--'text' is an implicit field emited by TikaEntityProcessor . Map it appropriately-->
<field column="text"/>
</entity>
<document>
</dataConfig>
{code}
With format=xml|html XPathEntityProcessor can be nested. This may help users extract more nested data from a file. It is even possible to create multiple documents from a single file
was (Author: noble.paul):
Let us provide a new TikaEntityProcessor
{code:xml}
<dataConfig>
<!-- use any of type DataSource<InputStream> -->
<dataSource type="BinURLDataSource"/>
<document>
<!-- The value of format can be text|xml|html . The implicit field 'text' will have that format.
default value is 'text' (if not specified) -->
<entity processor="TikaEntityProcessor" tikaConfig="tikaconfig.xml" url="${some.var.goes.here}" format="text">
<!--Do appropriate mapping here meta="true" means it is a metadata field -->
<field column="Author" meta="true" name="author"/>
<field column="title" meta="true" name="docTitle"/>
<!--'text' is an implicit field emited by TikaEntityProcessor . Map it appropriately-->
<field column="text"/>
</entity>
<document>
</dataConfig>
{code}
With format=xml|html XPathEntityProcessor can be nested. This may help users extract more nested data from a file. It is even possible to create multiple documents from a single file
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
> Attachments: SOLR-1358.patch, SOLR-1358.patch, SOLR-1358.patch
>
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1358) Integration of Solr Cell and
DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750855#action_12750855 ]
Noble Paul commented on SOLR-1358:
----------------------------------
Let us provide a new TikaEntityProcessor
{code:xml}
<entity processor="TikaEntityProcessor" tikaConfig="tikaconfig.xml" url="${some.var.goes.here}">
</entity>
{code}
This most likely would need a BinUrlDataSource/BinContentStreamDataSource because Tika uses binary inputs.
My suggestion is that TikaEntityProcessor live in the extraction contrib so that managing dependencies is easier. But we will have to make extraction have a compile-time dependency on DIH.
Grant , what do you think?
> Integration of Solr Cell and DataImportHandler
> ----------------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Akshay K. Ukey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Akshay K. Ukey updated SOLR-1358:
---------------------------------
Attachment: SOLR-1358.patch
Patch with test case, tika parser configurable via parser attribute for entity tag.
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
> Attachments: SOLR-1358.patch, SOLR-1358.patch, SOLR-1358.patch, SOLR-1358.patch
>
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noble Paul updated SOLR-1358:
-----------------------------
Comment: was deleted
(was: Configuration with attribute to select format of emitted content:
{code:xml}
<dataConfig>
<!-- use any of type DataSource<InputStream> -->
<dataSource type="BinURLDataSource"/>
<document>
<!-- 'emitFormat' can be one of text | html | xml -->
<entity processor="TikaEntityProcessor" tikaConfig="tikaconfig.xml" url="${some.var.goes.here}" emitFormat="xml" >
<!--Do appropriate mapping here meta="true" means it is a metadata field -->
<field column="Author" meta="true" name="author"/>
<field column="title" meta="true" name="docTitle"/>
<!--'text' is an implicit field emitted by TikaEntityProcessor . Map it appropriately-->
<field column="text"/>
</entity>
<document>
</dataConfig>
{code}
With 'emitFormat' different EntityProcessors can be chained. E.g. using "xml" value will allow chaining XPathEntityProcessor with TikaEntityProcessor for further custom processing.)
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
> Attachments: SOLR-1358.patch, SOLR-1358.patch
>
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Akshay K. Ukey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Akshay K. Ukey updated SOLR-1358:
---------------------------------
Attachment: SOLR-1358.patch
First cut patch. Not tested.
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
> Attachments: SOLR-1358.patch
>
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750855#action_12750855 ]
Noble Paul edited comment on SOLR-1358 at 12/8/09 3:29 PM:
-----------------------------------------------------------
Let us provide a new TikaEntityProcessor
{code:xml}
<dataConfig>
<!-- use any of type DataSource<InputStream> -->
<dataSource type="BinURLDataSource"/>
<document>
<!-- The value of format can be text|xml|html . The implicit field 'text' will have that format.
default value is 'text' (if not specified) -->
<entity processor="TikaEntityProcessor" tikaConfig="tikaconfig.xml" url="${some.var.goes.here}" format="text">
<!--Do appropriate mapping here meta="true" means it is a metadata field -->
<field column="Author" meta="true" name="author"/>
<field column="title" meta="true" name="docTitle"/>
<!--'text' is an implicit field emited by TikaEntityProcessor . Map it appropriately-->
<field column="text"/>
</entity>
<document>
</dataConfig>
{code}
With format=xml|html XPathEntityProcessor can be nested. This may help users extract more nested data from a file. It is even possible to create multiple documents from a single file
was (Author: noble.paul):
Let us provide a new TikaEntityProcessor
{code:xml}
<dataConfig>
<!-- use any of type DataSource<InputStream> -->
<dataSource type="BinURLDataSource"/>
<document>
<entity processor="TikaEntityProcessor" tikaConfig="tikaconfig.xml" url="${some.var.goes.here}">
<!--Do appropriate mapping here meta="true" means it is a metadata field -->
<field column="Author" meta="true" name="author"/>
<field column="title" meta="true" name="docTitle"/>
<!--'text' is an implicit field emited by TikaEntityProcessor . Map it appropriately-->
<field column="text"/>
</entity>
<document>
</dataConfig>
{code}
This most likely would need a BinUrlDataSource/BinContentStreamDataSource because Tika uses binary inputs.
My suggestion is that TikaEntityProcessor live in the extraction contrib so that managing dependencies is easier. But we will have to make extraction have a compile-time dependency on DIH.
Grant , what do you think?
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
> Attachments: SOLR-1358.patch, SOLR-1358.patch
>
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Akshay K. Ukey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789300#action_12789300 ]
Akshay K. Ukey edited comment on SOLR-1358 at 12/11/09 1:23 PM:
----------------------------------------------------------------
Patch with test case and with tika parser configurable via parser attribute for entity tag.
was (Author: akshay):
Patch with test case, tika parser configurable via parser attribute for entity tag.
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
> Attachments: SOLR-1358.patch, SOLR-1358.patch, SOLR-1358.patch, SOLR-1358.patch
>
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noble Paul updated SOLR-1358:
-----------------------------
Attachment: SOLR-1358.patch
onError implemented
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
> Attachments: SOLR-1358.patch, SOLR-1358.patch, SOLR-1358.patch
>
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Akshay K. Ukey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787466#action_12787466 ]
Akshay K. Ukey commented on SOLR-1358:
--------------------------------------
Configuration with attribute to select format of emitted content:
{code:xml}
<dataConfig>
<!-- use any of type DataSource<InputStream> -->
<dataSource type="BinURLDataSource"/>
<document>
<!-- 'emitFormat' can be one of text | html | xml -->
<entity processor="TikaEntityProcessor" tikaConfig="tikaconfig.xml" url="${some.var.goes.here}" emitFormat="xml" >
<!--Do appropriate mapping here meta="true" means it is a metadata field -->
<field column="Author" meta="true" name="author"/>
<field column="title" meta="true" name="docTitle"/>
<!--'text' is an implicit field emitted by TikaEntityProcessor . Map it appropriately-->
<field column="text"/>
</entity>
<document>
</dataConfig>
{code}
With 'emitFormat' different EntityProcessors can be chained. E.g. using "xml" value will allow chaining XPathEntityProcessor with TikaEntityProcessor for further custom processing.
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750855#action_12750855 ]
Noble Paul edited comment on SOLR-1358 at 12/8/09 8:50 AM:
-----------------------------------------------------------
Let us provide a new TikaEntityProcessor
{code:xml}
<dataConfig>
<!-- use any of type DataSource<InputStream> -->
<dataSource type="BinURLDataSource"/>
<document>
<entity processor="TikaEntityProcessor" tikaConfig="tikaconfig.xml" url="${some.var.goes.here}">
<!--Do appropriate mapping here meta="true" means it is a metadata field -->
<field column="Author" meta="true" name="author"/>
<field column="title" meta="true" name="docTitle"/>
<!--'text' is an implicit field emited by TikaEntityProcessor . Map it appropriately-->
<field column="text"/>
</entity>
<document>
</dataConfig>
{code}
This most likely would need a BinUrlDataSource/BinContentStreamDataSource because Tika uses binary inputs.
My suggestion is that TikaEntityProcessor live in the extraction contrib so that managing dependencies is easier. But we will have to make extraction have a compile-time dependency on DIH.
Grant , what do you think?
was (Author: noble.paul):
Let us provide a new TikaEntityProcessor
{code:xml}
<dataConfig>
<!-- use any of type DataSource<InputStream> -->
<dataSource type="BinURLDataSource"/>
<document>
<entity processor="TikaEntityProcessor" tikaConfig="tikaconfig.xml" url="${some.var.goes.here}">
</entity>
<document>
</dataConfig>
{code}
This most likely would need a BinUrlDataSource/BinContentStreamDataSource because Tika uses binary inputs.
My suggestion is that TikaEntityProcessor live in the extraction contrib so that managing dependencies is easier. But we will have to make extraction have a compile-time dependency on DIH.
Grant , what do you think?
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noble Paul reassigned SOLR-1358:
--------------------------------
Assignee: Noble Paul
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noble Paul updated SOLR-1358:
-----------------------------
Summary: Integration of Tika and DataImportHandler (was: Integration of Solr Cell and DataImportHandler)
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750855#action_12750855 ]
Noble Paul edited comment on SOLR-1358 at 12/8/09 6:36 AM:
-----------------------------------------------------------
Let us provide a new TikaEntityProcessor
{code:xml}
<dataConfig>
<!-- use any of type DataSource<InputStream> -->
<dataSource type="BinURLDataSource"/>
<document>
<entity processor="TikaEntityProcessor" tikaConfig="tikaconfig.xml" url="${some.var.goes.here}">
</entity>
<document>
</dataConfig>
{code}
This most likely would need a BinUrlDataSource/BinContentStreamDataSource because Tika uses binary inputs.
My suggestion is that TikaEntityProcessor live in the extraction contrib so that managing dependencies is easier. But we will have to make extraction have a compile-time dependency on DIH.
Grant , what do you think?
was (Author: noble.paul):
Let us provide a new TikaEntityProcessor
{code:xml}
<entity processor="TikaEntityProcessor" tikaConfig="tikaconfig.xml" url="${some.var.goes.here}">
</entity>
{code}
This most likely would need a BinUrlDataSource/BinContentStreamDataSource because Tika uses binary inputs.
My suggestion is that TikaEntityProcessor live in the extraction contrib so that managing dependencies is easier. But we will have to make extraction have a compile-time dependency on DIH.
Grant , what do you think?
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noble Paul updated SOLR-1358:
-----------------------------
Attachment: (was: SOLR-1358.patch)
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
> Attachments: SOLR-1358.patch, SOLR-1358.patch
>
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noble Paul resolved SOLR-1358.
------------------------------
Resolution: Fixed
Fix Version/s: 1.5
committed r889613
Thanks Akshay
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
> Fix For: 1.5
>
> Attachments: SOLR-1358.patch, SOLR-1358.patch, SOLR-1358.patch, SOLR-1358.patch
>
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1358) Integration of Tika and
DataImportHandler
Posted by "Akshay K. Ukey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788089#action_12788089 ]
Akshay K. Ukey commented on SOLR-1358:
--------------------------------------
Patch with fix for avoiding reading from data source continuously.
> Integration of Tika and DataImportHandler
> -----------------------------------------
>
> Key: SOLR-1358
> URL: https://issues.apache.org/jira/browse/SOLR-1358
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Sascha Szott
> Assignee: Noble Paul
> Attachments: SOLR-1358.patch, SOLR-1358.patch, SOLR-1358.patch
>
>
> At the moment, it's impossible to configure Solr such that it build up documents by using data that comes from both pdf documents and database table columns. Currently, to accomplish this task, it's up to the user to add some preprocessing that converts pdf files into plain text files. Therefore, I would like to see an integration of Solr Cell into DIH that makes those preprocessing obsolete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.