[jira] [Created] (TIKA-3835) parse cache - avoid re-parsing content that has not changed

["Nicholas DiPiazza (Jira)" <ji...@apache.org>]

Nicholas DiPiazza created TIKA-3835:
---------------------------------------

             Summary: parse cache - avoid re-parsing content that has not changed
                 Key: TIKA-3835
                 URL: https://issues.apache.org/jira/browse/TIKA-3835
             Project: Tika
          Issue Type: New Feature
          Components: tika-pipes
    Affects Versions: 2.2.0
            Reporter: Nicholas DiPiazza


Tika pipes should have an optional configuration to archive parsed results.

So the process would be 
 * pipe iterator has the next document: \{lastUpdated,docID}
 * if parse cache is enabled and parse cache contains \{lastUpdated,docID}
 ** Emit to the document to the emit queue and return.
 * Parse document
 * If parse cache is enabled, put into cache key=\{lastUpdated,docID}, value=\{document,metadata}

This will drastically improve full crawl times for customers using services especially cloud file services with strict rate limits.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)