Failing to detect the euro symbol, not-ascii character

[Shinichiro Abe <sh...@gmail.com>]

Hi,

I have some simple code to call Tika.
Tika 1.0/1.1 return "€100" but Tika 1.2 doesn't return "€100"(nothing):

      InputStream input = new ByteArrayInputStream("€100".getBytes("UTF-8"));
      Parser parser = new AutoDetectParser();
      ContentHandler contentHandler = new BodyContentHandler();
      ParseContext parseContext = new ParseContext();
      Metadata metadata = new Metadata();
      metadata.add("Content-Type", "application/octet-stream");
      parser.parse(input, contentHandler, metadata, parseContext);
      System.out.println(contentHandler.toString());

On Tika 1.2 when I replace the metadata with the following I can get "€100".

      metadata.add("Content-Type", "text/plain");

This behavior occurs not only in the case that it contains the euro symbol
but also in the case that it contains not-ascii unicode character.

I expect Tika 1.2 to detect like the behavior the same as Tika 1.0/1.1
which can detect the euro symbol even when Content-Type is application/octet-stream.

My question is whether this behavior of Tika 1.2 is correct, 
why Tika 1.2'behavior is changed, and whether I always need to give a right hint. 

Thanks in advance,
Shinichiro Abe

Re: Failing to detect the euro symbol, not-ascii character

[Shinichiro Abe <sh...@gmail.com>]

> Do you still have the issue if you supply some more text?
Yes, I do. I supplied more text in which it contains Japanese/Chinese character but Tika 1.2 could not detect this.

Thank you,
Shinichiro Abe

On 2012/11/15, at 1:59, Nick Burch wrote:

> On 14/11/12 09:17, Shinichiro Abe wrote:
>> Hi,
>> I have some simple code to call Tika.
>> Tika 1.0/1.1 return "€100" but Tika 1.2 doesn't return "€100"(nothing):
>> 
>>       InputStream input = new ByteArrayInputStream("€100".getBytes("UTF-8"));
> 
> 4 characters is a very short amount of data to be working on to detect. Do you still have the issue if you supply some more text?
> 
> Nick

Re: Failing to detect the euro symbol, not-ascii character

[Nick Burch <ap...@gagravarr.org>]

On 14/11/12 09:17, Shinichiro Abe wrote:
> Hi,
> 
> I have some simple code to call Tika.
> Tika 1.0/1.1 return "€100" but Tika 1.2 doesn't return "€100"(nothing):
>
>        InputStream input = new 
> ByteArrayInputStream("€100".getBytes("UTF-8"));

4 characters is a very short amount of data to be working on to detect. Do 
you still have the issue if you supply some more text?

Nick