RE: [bulk] Info from parser on handling partial input

[Hanssens Bart <Ba...@fedict.be>]

Hi Ken,


> The file was truncated because I'd configured Bixo to only fetch
> the first 65K of a file, to avoid problems caused by huge files.
...
> XML is not fine, from what I've seen - the parser will fail if it runs
> into the end of document before finishing the parse.

Using a StAX parser, I guess it should be possible to parse the XML
and just stop after running out of data at 64 K... 

> And binary formats like zip, pdf, etc are definitely not OK with
> truncation.

Have to check on PDF, but some data can probably be retrieved
(especially the "web-optimized" PDFs), 

Some zips might be OK: if one manages to get at least one zipentry
before hitting the 64 K limit (say xml-in-zip formats like ODF, OOXML,
ePUB), it should be possible to index it partially.

It will get messy, though, if the supporting libraries insist on checking
the completeness of the file...


Just my 2 cents,

Bart

Re: [bulk] Info from parser on handling partial input

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Thu, Oct 8, 2009 at 7:34 PM, Hanssens Bart <Ba...@fedict.be> wrote:
> Some zips might be OK: if one manages to get at least one zipentry
> before hitting the 64 K limit (say xml-in-zip formats like ODF, OOXML,
> ePUB), it should be possible to index it partially.

It's possible to read ZIP files in streaming mode, but see the caveats
listed in [1]. The current ZipParser in Tika does use the streaming
even though the result may be incorrect.

Once TIKA-153 is solved, we should be able to automatically switch to
more correct parsing when the full input document is available in
random-access mode.

[1] http://commons.apache.org/compress/zip.html

BR,

Jukka Zitting