You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Andrzej Bialecki (Created) (JIRA)" <ji...@apache.org> on 2012/02/17 01:09:59 UTC

[jira] [Created] (TIKA-863) MailContentHandler should not create AutoDetectParser on each call

MailContentHandler should not create AutoDetectParser on each call
------------------------------------------------------------------

                 Key: TIKA-863
                 URL: https://issues.apache.org/jira/browse/TIKA-863
             Project: Tika
          Issue Type: Bug
          Components: parser
    Affects Versions: 1.1
            Reporter: Andrzej Bialecki 


MailContentHandler is called from RFC822Parser, and it creates AutoDetectParser on each call to parse(...). The process or creating AutoDetectParser involves reading TikaConfig (not cached), which in turn involves parsing XML config files. Apart from the fact that this process is wasteful and heavy, in addition in a highly concurrent setup it leads to multiple threads blocking on SAX parser creation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (TIKA-863) MailContentHandler should not create AutoDetectParser on each call

Posted by "Andrzej Bialecki (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/TIKA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki  updated TIKA-863:
-----------------------------------

    Description: MailContentHandler is called from RFC822Parser, and it creates AutoDetectParser on each call to parse(...). The process of creating AutoDetectParser involves reading TikaConfig (not cached), which in turn involves parsing XML config files. Apart from the fact that this process is wasteful and heavy, in addition in a highly concurrent setup it leads to multiple threads blocking on SAX parser creation.  (was: MailContentHandler is called from RFC822Parser, and it creates AutoDetectParser on each call to parse(...). The process or creating AutoDetectParser involves reading TikaConfig (not cached), which in turn involves parsing XML config files. Apart from the fact that this process is wasteful and heavy, in addition in a highly concurrent setup it leads to multiple threads blocking on SAX parser creation.)
    
> MailContentHandler should not create AutoDetectParser on each call
> ------------------------------------------------------------------
>
>                 Key: TIKA-863
>                 URL: https://issues.apache.org/jira/browse/TIKA-863
>             Project: Tika
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.1
>            Reporter: Andrzej Bialecki 
>         Attachments: TIKA-863.patch
>
>
> MailContentHandler is called from RFC822Parser, and it creates AutoDetectParser on each call to parse(...). The process of creating AutoDetectParser involves reading TikaConfig (not cached), which in turn involves parsing XML config files. Apart from the fact that this process is wasteful and heavy, in addition in a highly concurrent setup it leads to multiple threads blocking on SAX parser creation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Resolved] (TIKA-863) MailContentHandler should not create AutoDetectParser on each call

Posted by "Nick Burch (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/TIKA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nick Burch resolved TIKA-863.
-----------------------------

       Resolution: Fixed
    Fix Version/s: 1.2
    
> MailContentHandler should not create AutoDetectParser on each call
> ------------------------------------------------------------------
>
>                 Key: TIKA-863
>                 URL: https://issues.apache.org/jira/browse/TIKA-863
>             Project: Tika
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.1
>            Reporter: Andrzej Bialecki 
>             Fix For: 1.2
>
>         Attachments: TIKA-863.patch
>
>
> MailContentHandler is called from RFC822Parser, and it creates AutoDetectParser on each call to parse(...). The process of creating AutoDetectParser involves reading TikaConfig (not cached), which in turn involves parsing XML config files. Apart from the fact that this process is wasteful and heavy, in addition in a highly concurrent setup it leads to multiple threads blocking on SAX parser creation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (TIKA-863) MailContentHandler should not create AutoDetectParser on each call

Posted by "Nick Burch (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/TIKA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404580#comment-13404580 ] 

Nick Burch commented on TIKA-863:
---------------------------------

Should be fixed in r1355782. We now avoid creating a new AutoDetectParser (and implicit TikaConfig) for each part in a RFC822 message. Instead, check for one on the ParseContext, otherwise cache the TikaConfig for the lifetime of the message being parsed, and grab the parser from that.
                
> MailContentHandler should not create AutoDetectParser on each call
> ------------------------------------------------------------------
>
>                 Key: TIKA-863
>                 URL: https://issues.apache.org/jira/browse/TIKA-863
>             Project: Tika
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.1
>            Reporter: Andrzej Bialecki 
>         Attachments: TIKA-863.patch
>
>
> MailContentHandler is called from RFC822Parser, and it creates AutoDetectParser on each call to parse(...). The process of creating AutoDetectParser involves reading TikaConfig (not cached), which in turn involves parsing XML config files. Apart from the fact that this process is wasteful and heavy, in addition in a highly concurrent setup it leads to multiple threads blocking on SAX parser creation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (TIKA-863) MailContentHandler should not create AutoDetectParser on each call

Posted by "Andrzej Bialecki (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/TIKA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210242#comment-13210242 ] 

Andrzej Bialecki  commented on TIKA-863:
----------------------------------------

This would work in my case, too. Though the same pattern would have to be applied in other places where Tika creates heavy parsers (not to mention the TikaConfig re-creation on each get ... quite costly), so I think a convenience method should be provided to avoid repeating the same logic in various places.
                
> MailContentHandler should not create AutoDetectParser on each call
> ------------------------------------------------------------------
>
>                 Key: TIKA-863
>                 URL: https://issues.apache.org/jira/browse/TIKA-863
>             Project: Tika
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.1
>            Reporter: Andrzej Bialecki 
>         Attachments: TIKA-863.patch
>
>
> MailContentHandler is called from RFC822Parser, and it creates AutoDetectParser on each call to parse(...). The process of creating AutoDetectParser involves reading TikaConfig (not cached), which in turn involves parsing XML config files. Apart from the fact that this process is wasteful and heavy, in addition in a highly concurrent setup it leads to multiple threads blocking on SAX parser creation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (TIKA-863) MailContentHandler should not create AutoDetectParser on each call

Posted by "Nick Burch (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/TIKA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210236#comment-13210236 ] 

Nick Burch commented on TIKA-863:
---------------------------------

I'm not sure what the best way is to provide an AutoDetectParser for this case though. 

MailContentHandler does seem to me to behave differently to cases like Zip, which have multiple sub-parts, and it is different to cases like word-with-excel

Maybe we should check for an AutoDetectParser on the context, use that if given, and if not lazy-create + cache a default one on the parser, and supply that through to the mail content handler?
                
> MailContentHandler should not create AutoDetectParser on each call
> ------------------------------------------------------------------
>
>                 Key: TIKA-863
>                 URL: https://issues.apache.org/jira/browse/TIKA-863
>             Project: Tika
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.1
>            Reporter: Andrzej Bialecki 
>         Attachments: TIKA-863.patch
>
>
> MailContentHandler is called from RFC822Parser, and it creates AutoDetectParser on each call to parse(...). The process of creating AutoDetectParser involves reading TikaConfig (not cached), which in turn involves parsing XML config files. Apart from the fact that this process is wasteful and heavy, in addition in a highly concurrent setup it leads to multiple threads blocking on SAX parser creation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (TIKA-863) MailContentHandler should not create AutoDetectParser on each call

Posted by "Andrzej Bialecki (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/TIKA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220812#comment-13220812 ] 

Andrzej Bialecki  commented on TIKA-863:
----------------------------------------

Nick, is there a chance of this making 1.1 ? Should I prepare another patch? though IIRC Jukka has changed in the meantime the way TikaConfig is inited...
                
> MailContentHandler should not create AutoDetectParser on each call
> ------------------------------------------------------------------
>
>                 Key: TIKA-863
>                 URL: https://issues.apache.org/jira/browse/TIKA-863
>             Project: Tika
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.1
>            Reporter: Andrzej Bialecki 
>         Attachments: TIKA-863.patch
>
>
> MailContentHandler is called from RFC822Parser, and it creates AutoDetectParser on each call to parse(...). The process of creating AutoDetectParser involves reading TikaConfig (not cached), which in turn involves parsing XML config files. Apart from the fact that this process is wasteful and heavy, in addition in a highly concurrent setup it leads to multiple threads blocking on SAX parser creation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (TIKA-863) MailContentHandler should not create AutoDetectParser on each call

Posted by "Nick Burch (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/TIKA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210205#comment-13210205 ] 

Nick Burch commented on TIKA-863:
---------------------------------

I'm not sure if we should be setting it as AutoDetectParser.class on the context or not. 

Part of me thinks that we should be setting it as Parser.class, if the user hasn't already set one. However, there is an issue that the parser set for that may be expecting something slightly different, eg an embedded document rather than effectively the core document, though you could argue that the mime contents are all embedded.

The second worry is with attaching additional objects to the ParseContext automatically. A user may choose to not set a recursing parser on the ParseContext, because they don't want embedded resources. If we add one, and the email contains a word document with an embedded excel spreadsheet, then suddenly because we added a Parser to the context recursion will trigger
                
> MailContentHandler should not create AutoDetectParser on each call
> ------------------------------------------------------------------
>
>                 Key: TIKA-863
>                 URL: https://issues.apache.org/jira/browse/TIKA-863
>             Project: Tika
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.1
>            Reporter: Andrzej Bialecki 
>         Attachments: TIKA-863.patch
>
>
> MailContentHandler is called from RFC822Parser, and it creates AutoDetectParser on each call to parse(...). The process of creating AutoDetectParser involves reading TikaConfig (not cached), which in turn involves parsing XML config files. Apart from the fact that this process is wasteful and heavy, in addition in a highly concurrent setup it leads to multiple threads blocking on SAX parser creation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (TIKA-863) MailContentHandler should not create AutoDetectParser on each call

Posted by "Nick Burch (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/TIKA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210245#comment-13210245 ] 

Nick Burch commented on TIKA-863:
---------------------------------

We could check for the TikaConfig on the ParseContext, which if present could be used when creating parsers / detectors etc inside Parsers?
                
> MailContentHandler should not create AutoDetectParser on each call
> ------------------------------------------------------------------
>
>                 Key: TIKA-863
>                 URL: https://issues.apache.org/jira/browse/TIKA-863
>             Project: Tika
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.1
>            Reporter: Andrzej Bialecki 
>         Attachments: TIKA-863.patch
>
>
> MailContentHandler is called from RFC822Parser, and it creates AutoDetectParser on each call to parse(...). The process of creating AutoDetectParser involves reading TikaConfig (not cached), which in turn involves parsing XML config files. Apart from the fact that this process is wasteful and heavy, in addition in a highly concurrent setup it leads to multiple threads blocking on SAX parser creation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (TIKA-863) MailContentHandler should not create AutoDetectParser on each call

Posted by "Andrzej Bialecki (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/TIKA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210221#comment-13210221 ] 

Andrzej Bialecki  commented on TIKA-863:
----------------------------------------

I see your point, it makes sense - no hidden surprises. However, I think the part of the patch that simply checks whether an existing AutoDetectParser has been provided and uses it if present, is still useful and makes a huge difference.
                
> MailContentHandler should not create AutoDetectParser on each call
> ------------------------------------------------------------------
>
>                 Key: TIKA-863
>                 URL: https://issues.apache.org/jira/browse/TIKA-863
>             Project: Tika
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.1
>            Reporter: Andrzej Bialecki 
>         Attachments: TIKA-863.patch
>
>
> MailContentHandler is called from RFC822Parser, and it creates AutoDetectParser on each call to parse(...). The process of creating AutoDetectParser involves reading TikaConfig (not cached), which in turn involves parsing XML config files. Apart from the fact that this process is wasteful and heavy, in addition in a highly concurrent setup it leads to multiple threads blocking on SAX parser creation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (TIKA-863) MailContentHandler should not create AutoDetectParser on each call

Posted by "Andrzej Bialecki (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/TIKA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki  updated TIKA-863:
-----------------------------------

    Attachment: TIKA-863.patch

Patch to address this issue. AutoDetectParser instance is cached in ParseContext.

All tests pass with this patch. An application that I'm testing with this patch experienced a 12x speedup in mail parsing.
                
> MailContentHandler should not create AutoDetectParser on each call
> ------------------------------------------------------------------
>
>                 Key: TIKA-863
>                 URL: https://issues.apache.org/jira/browse/TIKA-863
>             Project: Tika
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.1
>            Reporter: Andrzej Bialecki 
>         Attachments: TIKA-863.patch
>
>
> MailContentHandler is called from RFC822Parser, and it creates AutoDetectParser on each call to parse(...). The process or creating AutoDetectParser involves reading TikaConfig (not cached), which in turn involves parsing XML config files. Apart from the fact that this process is wasteful and heavy, in addition in a highly concurrent setup it leads to multiple threads blocking on SAX parser creation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira