You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@creadur.apache.org by "Marshall Schor (JIRA)" <ji...@apache.org> on 2010/12/10 04:45:00 UTC

[jira] Created: (RAT-81) MalformedInputException thrown when RAT tries reading file

MalformedInputException thrown when RAT tries reading file
----------------------------------------------------------

                 Key: RAT-81
                 URL: https://issues.apache.org/jira/browse/RAT-81
             Project: RAT
          Issue Type: Bug
    Affects Versions: 0.6
         Environment: Linux (Ubuntu) on x86, running with "default" file encoding set to UTF-8
            Reporter: Marshall Schor
            Priority: Minor


To reproduce, set the platform default locale to something that indicates UTF-8 file encoding.
This causes code in (for example) org.apache.rat.document.impl.FileDocument which return FileReader to set up RAT to use a reader which is using the platform default character encoding (in this case UTF-8).

If the file being processed is not encoded in this , it is possible that the reader will read some data which is "invalid" UTF-8 encodings, which causes the reader to throw a MalformedInputException error.

One case we found:

The file being examined had invalid UTF-8 encodings.  First, Rat ran the BinaryGuesser - but that returned false because it attempted to read the first 100 or so chars, and got a "MalformedInputException" instead, so the try/catch block just ended up returning "false" (not binary).  Then the HeaderChecker tried to read the file to check the header, and got this same exception - but this time, it made RAT fail.

Here's the last part of the stack trace:

Caused by: org.apache.rat.report.RatReportFailedException: Analysis failed
    at org.apache.rat.report.xml.XmlReport.report(XmlReport.java:66)
    at org.apache.rat.mp.FilesReportable.run(FilesReportable.java:69)
    at org.apache.rat.Report.report(Report.java:292)
    at org.apache.rat.Report.report(Report.java:272)
    at org.apache.rat.mp.AbstractRatMojo.createReport(AbstractRatMojo.java:341)
    ... 23 more
Caused by: org.apache.rat.document.RatDocumentAnalysisException: Cannot analyse header
    at org.apache.rat.report.analyser.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:54)
    at org.apache.rat.document.impl.util.DocumentAnalyserMultiplexer.analyse(DocumentAnalyserMultiplexer.java:37)
    at org.apache.rat.document.impl.util.ConditionalAnalyser.matches(ConditionalAnalyser.java:44)
    at org.apache.rat.document.impl.util.ConditionalAnalyser.analyse(ConditionalAnalyser.java:50)
    at org.apache.rat.report.xml.XmlReport.report(XmlReport.java:64)
    ... 27 more
Caused by: org.apache.rat.analysis.RatHeaderAnalysisException: Cannot read header for /home/tgoetz/tmp/uimaj-2.3.1/uimaj-core/src/test/resources/pearTests/encodingTests/UTF16_with_signature.xml
    at org.apache.rat.report.analyser.HeaderCheckWorker.read(HeaderCheckWorker.java:96)
    at org.apache.rat.report.analyser.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:50)
    ... 31 more
Caused by: sun.io.MalformedInputException
    at sun.io.ByteToCharUTF8.convert(ByteToCharUTF8.java:294)
    at sun.nio.cs.StreamDecoder$ConverterSD.convertInto(StreamDecoder.java:316)
    at sun.nio.cs.StreamDecoder$ConverterSD.implRead(StreamDecoder.java:366)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:252)
    at java.io.InputStreamReader.read(InputStreamReader.java:212)
    at java.io.BufferedReader.fill(BufferedReader.java:157)
    at java.io.BufferedReader.readLine(BufferedReader.java:320)
    at java.io.BufferedReader.readLine(BufferedReader.java:383)
    at org.apache.rat.report.analyser.HeaderCheckWorker.readLine(HeaderCheckWorker.java:111)
    at org.apache.rat.report.analyser.HeaderCheckWorker.read(HeaderCheckWorker.java:89)
    ... 32 more 

Work-around: mark these files for explicit exclusion.

Fix: change the binaryguesser to read the files in binary (not assuming any character coding) and operate with that data.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] [Updated] (RAT-81) MalformedInputException thrown when RAT tries reading file

Posted by "Stefan Bodewig (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/RAT-81?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stefan Bodewig updated RAT-81:
------------------------------

          Component/s: engine
    Affects Version/s: 0.7
        Fix Version/s: 0.8
             Assignee: Stefan Bodewig

> MalformedInputException thrown when RAT tries reading file
> ----------------------------------------------------------
>
>                 Key: RAT-81
>                 URL: https://issues.apache.org/jira/browse/RAT-81
>             Project: RAT
>          Issue Type: Bug
>          Components: engine
>    Affects Versions: 0.6, 0.7
>         Environment: Linux (Ubuntu) on x86, running with "default" file encoding set to UTF-8
>            Reporter: Marshall Schor
>            Assignee: Stefan Bodewig
>            Priority: Minor
>             Fix For: 0.8
>
>
> To reproduce, set the platform default locale to something that indicates UTF-8 file encoding.
> This causes code in (for example) org.apache.rat.document.impl.FileDocument which return FileReader to set up RAT to use a reader which is using the platform default character encoding (in this case UTF-8).
> If the file being processed is not encoded in this , it is possible that the reader will read some data which is "invalid" UTF-8 encodings, which causes the reader to throw a MalformedInputException error.
> One case we found:
> The file being examined had invalid UTF-8 encodings.  First, Rat ran the BinaryGuesser - but that returned false because it attempted to read the first 100 or so chars, and got a "MalformedInputException" instead, so the try/catch block just ended up returning "false" (not binary).  Then the HeaderChecker tried to read the file to check the header, and got this same exception - but this time, it made RAT fail.
> Here's the last part of the stack trace:
> Caused by: org.apache.rat.report.RatReportFailedException: Analysis failed
>     at org.apache.rat.report.xml.XmlReport.report(XmlReport.java:66)
>     at org.apache.rat.mp.FilesReportable.run(FilesReportable.java:69)
>     at org.apache.rat.Report.report(Report.java:292)
>     at org.apache.rat.Report.report(Report.java:272)
>     at org.apache.rat.mp.AbstractRatMojo.createReport(AbstractRatMojo.java:341)
>     ... 23 more
> Caused by: org.apache.rat.document.RatDocumentAnalysisException: Cannot analyse header
>     at org.apache.rat.report.analyser.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:54)
>     at org.apache.rat.document.impl.util.DocumentAnalyserMultiplexer.analyse(DocumentAnalyserMultiplexer.java:37)
>     at org.apache.rat.document.impl.util.ConditionalAnalyser.matches(ConditionalAnalyser.java:44)
>     at org.apache.rat.document.impl.util.ConditionalAnalyser.analyse(ConditionalAnalyser.java:50)
>     at org.apache.rat.report.xml.XmlReport.report(XmlReport.java:64)
>     ... 27 more
> Caused by: org.apache.rat.analysis.RatHeaderAnalysisException: Cannot read header for /home/tgoetz/tmp/uimaj-2.3.1/uimaj-core/src/test/resources/pearTests/encodingTests/UTF16_with_signature.xml
>     at org.apache.rat.report.analyser.HeaderCheckWorker.read(HeaderCheckWorker.java:96)
>     at org.apache.rat.report.analyser.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:50)
>     ... 31 more
> Caused by: sun.io.MalformedInputException
>     at sun.io.ByteToCharUTF8.convert(ByteToCharUTF8.java:294)
>     at sun.nio.cs.StreamDecoder$ConverterSD.convertInto(StreamDecoder.java:316)
>     at sun.nio.cs.StreamDecoder$ConverterSD.implRead(StreamDecoder.java:366)
>     at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:252)
>     at java.io.InputStreamReader.read(InputStreamReader.java:212)
>     at java.io.BufferedReader.fill(BufferedReader.java:157)
>     at java.io.BufferedReader.readLine(BufferedReader.java:320)
>     at java.io.BufferedReader.readLine(BufferedReader.java:383)
>     at org.apache.rat.report.analyser.HeaderCheckWorker.readLine(HeaderCheckWorker.java:111)
>     at org.apache.rat.report.analyser.HeaderCheckWorker.read(HeaderCheckWorker.java:89)
>     ... 32 more 
> Work-around: mark these files for explicit exclusion.
> Fix: change the binaryguesser to read the files in binary (not assuming any character coding) and operate with that data.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (RAT-81) MalformedInputException thrown when RAT tries reading file

Posted by "Stefan Bodewig (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/RAT-81?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066586#comment-13066586 ] 

Stefan Bodewig commented on RAT-81:
-----------------------------------

svn revision 1147525 now contains the NIO based code which I think should mark the files as binary for you.  It would be good if you could verify for me whether RAT's trunk behaves as desired now.

> MalformedInputException thrown when RAT tries reading file
> ----------------------------------------------------------
>
>                 Key: RAT-81
>                 URL: https://issues.apache.org/jira/browse/RAT-81
>             Project: RAT
>          Issue Type: Bug
>          Components: engine
>    Affects Versions: 0.6, 0.7
>         Environment: Linux (Ubuntu) on x86, running with "default" file encoding set to UTF-8
>            Reporter: Marshall Schor
>            Assignee: Stefan Bodewig
>            Priority: Minor
>             Fix For: 0.8
>
>
> To reproduce, set the platform default locale to something that indicates UTF-8 file encoding.
> This causes code in (for example) org.apache.rat.document.impl.FileDocument which return FileReader to set up RAT to use a reader which is using the platform default character encoding (in this case UTF-8).
> If the file being processed is not encoded in this , it is possible that the reader will read some data which is "invalid" UTF-8 encodings, which causes the reader to throw a MalformedInputException error.
> One case we found:
> The file being examined had invalid UTF-8 encodings.  First, Rat ran the BinaryGuesser - but that returned false because it attempted to read the first 100 or so chars, and got a "MalformedInputException" instead, so the try/catch block just ended up returning "false" (not binary).  Then the HeaderChecker tried to read the file to check the header, and got this same exception - but this time, it made RAT fail.
> Here's the last part of the stack trace:
> Caused by: org.apache.rat.report.RatReportFailedException: Analysis failed
>     at org.apache.rat.report.xml.XmlReport.report(XmlReport.java:66)
>     at org.apache.rat.mp.FilesReportable.run(FilesReportable.java:69)
>     at org.apache.rat.Report.report(Report.java:292)
>     at org.apache.rat.Report.report(Report.java:272)
>     at org.apache.rat.mp.AbstractRatMojo.createReport(AbstractRatMojo.java:341)
>     ... 23 more
> Caused by: org.apache.rat.document.RatDocumentAnalysisException: Cannot analyse header
>     at org.apache.rat.report.analyser.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:54)
>     at org.apache.rat.document.impl.util.DocumentAnalyserMultiplexer.analyse(DocumentAnalyserMultiplexer.java:37)
>     at org.apache.rat.document.impl.util.ConditionalAnalyser.matches(ConditionalAnalyser.java:44)
>     at org.apache.rat.document.impl.util.ConditionalAnalyser.analyse(ConditionalAnalyser.java:50)
>     at org.apache.rat.report.xml.XmlReport.report(XmlReport.java:64)
>     ... 27 more
> Caused by: org.apache.rat.analysis.RatHeaderAnalysisException: Cannot read header for /home/tgoetz/tmp/uimaj-2.3.1/uimaj-core/src/test/resources/pearTests/encodingTests/UTF16_with_signature.xml
>     at org.apache.rat.report.analyser.HeaderCheckWorker.read(HeaderCheckWorker.java:96)
>     at org.apache.rat.report.analyser.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:50)
>     ... 31 more
> Caused by: sun.io.MalformedInputException
>     at sun.io.ByteToCharUTF8.convert(ByteToCharUTF8.java:294)
>     at sun.nio.cs.StreamDecoder$ConverterSD.convertInto(StreamDecoder.java:316)
>     at sun.nio.cs.StreamDecoder$ConverterSD.implRead(StreamDecoder.java:366)
>     at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:252)
>     at java.io.InputStreamReader.read(InputStreamReader.java:212)
>     at java.io.BufferedReader.fill(BufferedReader.java:157)
>     at java.io.BufferedReader.readLine(BufferedReader.java:320)
>     at java.io.BufferedReader.readLine(BufferedReader.java:383)
>     at org.apache.rat.report.analyser.HeaderCheckWorker.readLine(HeaderCheckWorker.java:111)
>     at org.apache.rat.report.analyser.HeaderCheckWorker.read(HeaderCheckWorker.java:89)
>     ... 32 more 
> Work-around: mark these files for explicit exclusion.
> Fix: change the binaryguesser to read the files in binary (not assuming any character coding) and operate with that data.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (RAT-81) MalformedInputException thrown when RAT tries reading file

Posted by "Stefan Bodewig (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/RAT-81?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stefan Bodewig resolved RAT-81.
-------------------------------

    Resolution: Fixed

Let's be optimistic and assume it is fixed, please reopen if the problem pops up again once 0.8 has been released.

> MalformedInputException thrown when RAT tries reading file
> ----------------------------------------------------------
>
>                 Key: RAT-81
>                 URL: https://issues.apache.org/jira/browse/RAT-81
>             Project: RAT
>          Issue Type: Bug
>          Components: engine
>    Affects Versions: 0.6, 0.7
>         Environment: Linux (Ubuntu) on x86, running with "default" file encoding set to UTF-8
>            Reporter: Marshall Schor
>            Assignee: Stefan Bodewig
>            Priority: Minor
>             Fix For: 0.8
>
>
> To reproduce, set the platform default locale to something that indicates UTF-8 file encoding.
> This causes code in (for example) org.apache.rat.document.impl.FileDocument which return FileReader to set up RAT to use a reader which is using the platform default character encoding (in this case UTF-8).
> If the file being processed is not encoded in this , it is possible that the reader will read some data which is "invalid" UTF-8 encodings, which causes the reader to throw a MalformedInputException error.
> One case we found:
> The file being examined had invalid UTF-8 encodings.  First, Rat ran the BinaryGuesser - but that returned false because it attempted to read the first 100 or so chars, and got a "MalformedInputException" instead, so the try/catch block just ended up returning "false" (not binary).  Then the HeaderChecker tried to read the file to check the header, and got this same exception - but this time, it made RAT fail.
> Here's the last part of the stack trace:
> Caused by: org.apache.rat.report.RatReportFailedException: Analysis failed
>     at org.apache.rat.report.xml.XmlReport.report(XmlReport.java:66)
>     at org.apache.rat.mp.FilesReportable.run(FilesReportable.java:69)
>     at org.apache.rat.Report.report(Report.java:292)
>     at org.apache.rat.Report.report(Report.java:272)
>     at org.apache.rat.mp.AbstractRatMojo.createReport(AbstractRatMojo.java:341)
>     ... 23 more
> Caused by: org.apache.rat.document.RatDocumentAnalysisException: Cannot analyse header
>     at org.apache.rat.report.analyser.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:54)
>     at org.apache.rat.document.impl.util.DocumentAnalyserMultiplexer.analyse(DocumentAnalyserMultiplexer.java:37)
>     at org.apache.rat.document.impl.util.ConditionalAnalyser.matches(ConditionalAnalyser.java:44)
>     at org.apache.rat.document.impl.util.ConditionalAnalyser.analyse(ConditionalAnalyser.java:50)
>     at org.apache.rat.report.xml.XmlReport.report(XmlReport.java:64)
>     ... 27 more
> Caused by: org.apache.rat.analysis.RatHeaderAnalysisException: Cannot read header for /home/tgoetz/tmp/uimaj-2.3.1/uimaj-core/src/test/resources/pearTests/encodingTests/UTF16_with_signature.xml
>     at org.apache.rat.report.analyser.HeaderCheckWorker.read(HeaderCheckWorker.java:96)
>     at org.apache.rat.report.analyser.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:50)
>     ... 31 more
> Caused by: sun.io.MalformedInputException
>     at sun.io.ByteToCharUTF8.convert(ByteToCharUTF8.java:294)
>     at sun.nio.cs.StreamDecoder$ConverterSD.convertInto(StreamDecoder.java:316)
>     at sun.nio.cs.StreamDecoder$ConverterSD.implRead(StreamDecoder.java:366)
>     at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:252)
>     at java.io.InputStreamReader.read(InputStreamReader.java:212)
>     at java.io.BufferedReader.fill(BufferedReader.java:157)
>     at java.io.BufferedReader.readLine(BufferedReader.java:320)
>     at java.io.BufferedReader.readLine(BufferedReader.java:383)
>     at org.apache.rat.report.analyser.HeaderCheckWorker.readLine(HeaderCheckWorker.java:111)
>     at org.apache.rat.report.analyser.HeaderCheckWorker.read(HeaderCheckWorker.java:89)
>     ... 32 more 
> Work-around: mark these files for explicit exclusion.
> Fix: change the binaryguesser to read the files in binary (not assuming any character coding) and operate with that data.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (RAT-81) MalformedInputException thrown when RAT tries reading file

Posted by "Stefan Bodewig (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/RAT-81?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065883#comment-13065883 ] 

Stefan Bodewig commented on RAT-81:
-----------------------------------

I think it depends on the java VM as well, I don't seem to get the MalformedInputException on my Ubuntu box using OpenJDK6.

svn revision 1147110 contains an attempt at a testcase but it cannot assert anything since the precondition (there is a MIE) is never true for me.

I think I know how to fix the problem using NIO but so far am reluctant as I don't have a good way to regression test it.



> MalformedInputException thrown when RAT tries reading file
> ----------------------------------------------------------
>
>                 Key: RAT-81
>                 URL: https://issues.apache.org/jira/browse/RAT-81
>             Project: RAT
>          Issue Type: Bug
>          Components: engine
>    Affects Versions: 0.6, 0.7
>         Environment: Linux (Ubuntu) on x86, running with "default" file encoding set to UTF-8
>            Reporter: Marshall Schor
>            Assignee: Stefan Bodewig
>            Priority: Minor
>             Fix For: 0.8
>
>
> To reproduce, set the platform default locale to something that indicates UTF-8 file encoding.
> This causes code in (for example) org.apache.rat.document.impl.FileDocument which return FileReader to set up RAT to use a reader which is using the platform default character encoding (in this case UTF-8).
> If the file being processed is not encoded in this , it is possible that the reader will read some data which is "invalid" UTF-8 encodings, which causes the reader to throw a MalformedInputException error.
> One case we found:
> The file being examined had invalid UTF-8 encodings.  First, Rat ran the BinaryGuesser - but that returned false because it attempted to read the first 100 or so chars, and got a "MalformedInputException" instead, so the try/catch block just ended up returning "false" (not binary).  Then the HeaderChecker tried to read the file to check the header, and got this same exception - but this time, it made RAT fail.
> Here's the last part of the stack trace:
> Caused by: org.apache.rat.report.RatReportFailedException: Analysis failed
>     at org.apache.rat.report.xml.XmlReport.report(XmlReport.java:66)
>     at org.apache.rat.mp.FilesReportable.run(FilesReportable.java:69)
>     at org.apache.rat.Report.report(Report.java:292)
>     at org.apache.rat.Report.report(Report.java:272)
>     at org.apache.rat.mp.AbstractRatMojo.createReport(AbstractRatMojo.java:341)
>     ... 23 more
> Caused by: org.apache.rat.document.RatDocumentAnalysisException: Cannot analyse header
>     at org.apache.rat.report.analyser.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:54)
>     at org.apache.rat.document.impl.util.DocumentAnalyserMultiplexer.analyse(DocumentAnalyserMultiplexer.java:37)
>     at org.apache.rat.document.impl.util.ConditionalAnalyser.matches(ConditionalAnalyser.java:44)
>     at org.apache.rat.document.impl.util.ConditionalAnalyser.analyse(ConditionalAnalyser.java:50)
>     at org.apache.rat.report.xml.XmlReport.report(XmlReport.java:64)
>     ... 27 more
> Caused by: org.apache.rat.analysis.RatHeaderAnalysisException: Cannot read header for /home/tgoetz/tmp/uimaj-2.3.1/uimaj-core/src/test/resources/pearTests/encodingTests/UTF16_with_signature.xml
>     at org.apache.rat.report.analyser.HeaderCheckWorker.read(HeaderCheckWorker.java:96)
>     at org.apache.rat.report.analyser.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:50)
>     ... 31 more
> Caused by: sun.io.MalformedInputException
>     at sun.io.ByteToCharUTF8.convert(ByteToCharUTF8.java:294)
>     at sun.nio.cs.StreamDecoder$ConverterSD.convertInto(StreamDecoder.java:316)
>     at sun.nio.cs.StreamDecoder$ConverterSD.implRead(StreamDecoder.java:366)
>     at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:252)
>     at java.io.InputStreamReader.read(InputStreamReader.java:212)
>     at java.io.BufferedReader.fill(BufferedReader.java:157)
>     at java.io.BufferedReader.readLine(BufferedReader.java:320)
>     at java.io.BufferedReader.readLine(BufferedReader.java:383)
>     at org.apache.rat.report.analyser.HeaderCheckWorker.readLine(HeaderCheckWorker.java:111)
>     at org.apache.rat.report.analyser.HeaderCheckWorker.read(HeaderCheckWorker.java:89)
>     ... 32 more 
> Work-around: mark these files for explicit exclusion.
> Fix: change the binaryguesser to read the files in binary (not assuming any character coding) and operate with that data.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira