You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by "Jukka Zitting (JIRA)" <ji...@apache.org> on 2011/01/18 16:40:43 UTC

[jira] Created: (JCR-2864) Use out-of-process text extraction

Use out-of-process text extraction
----------------------------------

                 Key: JCR-2864
                 URL: https://issues.apache.org/jira/browse/JCR-2864
             Project: Jackrabbit Content Repository
          Issue Type: Improvement
          Components: indexing, jackrabbit-core
            Reporter: Jukka Zitting
            Assignee: Jukka Zitting
             Fix For: 2.3.0


The upcoming Tika 0.9 release will contain a highly useful out-of-process text extraction feature (TIKA-416) that we should use also in Jackrabbit.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (JCR-2864) Use out-of-process text extraction

Posted by "Sébastien Launay (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/JCR-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12985603#action_12985603 ] 

Sébastien Launay commented on JCR-2864:
---------------------------------------

Yes it definitely needs to be handle in Tika for other projects using it.
We happened to have this issue in a webapp (1 tomcat with 1 webapp consuming 60% of RAM+Swap and -Xmx == -Xms) and a shell script for launching custom data processing executed from the JVM with ProcessBuilder.
I guess this can become problematic when we have an enterprise server with multiple webapps (one or several using JR/Tika).

> Use out-of-process text extraction
> ----------------------------------
>
>                 Key: JCR-2864
>                 URL: https://issues.apache.org/jira/browse/JCR-2864
>             Project: Jackrabbit Content Repository
>          Issue Type: Improvement
>          Components: indexing, jackrabbit-core
>            Reporter: Jukka Zitting
>            Assignee: Jukka Zitting
>             Fix For: 2.3.0
>
>
> The upcoming Tika 0.9 release will contain a highly useful out-of-process text extraction feature (TIKA-416) that we should use also in Jackrabbit.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (JCR-2864) Use out-of-process text extraction

Posted by "Sébastien Launay (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/JCR-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12985577#action_12985577 ] 

Sébastien Launay commented on JCR-2864:
---------------------------------------

I like this isolation because we can also kill parsers blocked into a infinite loop.

But there might be an issue at least on GNU/Linux systems (vanilla kernel) because of how processes are created.
Indeed, creating a process required the same amount of free memory as the memory consumed by the parent process (fork() then exec()).
This can be a big issue for enterprise server where a lot of memory is dedicated to the JVM and creating subprocesses would result in the system swapping or error like "Cannot run program "java": java.io.IOException: error=12, Cannot allocate memory".

A solution often recommended is to start one light process parallel to the main one and communicate with it (e.g. by socket) to create the subprocesses.

More information can be found here:
http://developers.sun.com/solaris/articles/subprocess/subprocess.html


> Use out-of-process text extraction
> ----------------------------------
>
>                 Key: JCR-2864
>                 URL: https://issues.apache.org/jira/browse/JCR-2864
>             Project: Jackrabbit Content Repository
>          Issue Type: Improvement
>          Components: indexing, jackrabbit-core
>            Reporter: Jukka Zitting
>            Assignee: Jukka Zitting
>             Fix For: 2.3.0
>
>
> The upcoming Tika 0.9 release will contain a highly useful out-of-process text extraction feature (TIKA-416) that we should use also in Jackrabbit.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] [Resolved] (JCR-2864) Use out-of-process text extraction

Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/JCR-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jukka Zitting resolved JCR-2864.
--------------------------------

    Resolution: Fixed

I've implemented this by adding a new "forkJavaCommand" configuration option. When specified, this command will be used to fork background text extraction processes up to the configured "extractorPoolSize" limit (by default 2 x processor count). The pool size limit will only become effective once we upgrade to Tika 1.0 where TIKA-639 is implemented.

The fork memory use issue is best handled within the Tika issue.

> Use out-of-process text extraction
> ----------------------------------
>
>                 Key: JCR-2864
>                 URL: https://issues.apache.org/jira/browse/JCR-2864
>             Project: Jackrabbit Content Repository
>          Issue Type: Improvement
>          Components: indexing, jackrabbit-core
>            Reporter: Jukka Zitting
>            Assignee: Jukka Zitting
>             Fix For: 2.3.0
>
>
> The upcoming Tika 0.9 release will contain a highly useful out-of-process text extraction feature (TIKA-416) that we should use also in Jackrabbit.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (JCR-2864) Use out-of-process text extraction

Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/JCR-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12985591#action_12985591 ] 

Jukka Zitting commented on JCR-2864:
------------------------------------

Hmm, good point about subprocess creation. That should probably be handled in Tika though I'm not sure where the proper tradeoff between complexity and fail-safety lies. Can we somehow estimate how likely the described fork() memory issues would be in Jackrabbit deployments.

> Use out-of-process text extraction
> ----------------------------------
>
>                 Key: JCR-2864
>                 URL: https://issues.apache.org/jira/browse/JCR-2864
>             Project: Jackrabbit Content Repository
>          Issue Type: Improvement
>          Components: indexing, jackrabbit-core
>            Reporter: Jukka Zitting
>            Assignee: Jukka Zitting
>             Fix For: 2.3.0
>
>
> The upcoming Tika 0.9 release will contain a highly useful out-of-process text extraction feature (TIKA-416) that we should use also in Jackrabbit.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (JCR-2864) Use out-of-process text extraction

Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/JCR-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988716#comment-12988716 ] 

Jukka Zitting commented on JCR-2864:
------------------------------------

I filed TIKA-591 for tracking this within Tika. See also HADOOP-5059 for a similar problem in the Hadoop land.

> Use out-of-process text extraction
> ----------------------------------
>
>                 Key: JCR-2864
>                 URL: https://issues.apache.org/jira/browse/JCR-2864
>             Project: Jackrabbit Content Repository
>          Issue Type: Improvement
>          Components: indexing, jackrabbit-core
>            Reporter: Jukka Zitting
>            Assignee: Jukka Zitting
>             Fix For: 2.3.0
>
>
> The upcoming Tika 0.9 release will contain a highly useful out-of-process text extraction feature (TIKA-416) that we should use also in Jackrabbit.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira