[incubator-daffodil] branch master updated: Switch to scala-xml EncodingHeuristics to determine schema encoding

[sl...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

slawrence pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-daffodil.git


The following commit(s) were added to refs/heads/master by this push:
     new 7d96b4c  Switch to scala-xml EncodingHeuristics to determine schema encoding
7d96b4c is described below

commit 7d96b4cbae3949f266674da0a44e4909f565759f
Author: Steve Lawrence <sl...@apache.org>
AuthorDate: Fri Dec 20 09:15:38 2019 -0500

    Switch to scala-xml EncodingHeuristics to determine schema encoding
    
    There seems to be a bug or race condigion in Apache Commons IO
    XmlStreamReader that sporadically fails to detect UTF-16 encoded
    schmeas, which causes Daffodil to fail to parse the schemas. Though it's
    difficult reliably reproduce.
    
    Try switching to scala-xml's EncodingHeuristics to determine the schema
    encoding and see if that fixes the problem. If not, it's likely a bug in
    Daffodil somewhere.
    
    DAFFODIL-1908
---
 .../daffodil/xml/DaffodilConstructingLoader.scala  | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/daffodil-lib/src/main/scala/org/apache/daffodil/xml/DaffodilConstructingLoader.scala b/daffodil-lib/src/main/scala/org/apache/daffodil/xml/DaffodilConstructingLoader.scala
index 5f86300..111e58b 100644
--- a/daffodil-lib/src/main/scala/org/apache/daffodil/xml/DaffodilConstructingLoader.scala
+++ b/daffodil-lib/src/main/scala/org/apache/daffodil/xml/DaffodilConstructingLoader.scala
@@ -17,11 +17,14 @@
 
 package org.apache.daffodil.xml
 
-import scala.xml.parsing.ConstructingParser
-import scala.xml._
+import java.io.BufferedInputStream
 import java.net.URI
+
 import scala.io.Source
-import org.apache.commons.io.input.XmlStreamReader
+import scala.xml._
+import scala.xml.include.sax.EncodingHeuristics
+import scala.xml.parsing.ConstructingParser
+
 import org.apache.daffodil.exceptions.Assert
 
 /**
@@ -74,16 +77,13 @@ object Position {
  */
 class DaffodilConstructingLoader(uri: URI, errorHandler: org.xml.sax.ErrorHandler)
   extends ConstructingParser({
-    //
     // Note: we must open the XML carefully since it might be in some non
     // default encoding (we have tests that have UTF-16 for example)
-    //
-    val is = uri.toURL.openStream()
-    val rdr = new XmlStreamReader(is) // apache has a good thing for determining XML charset
-    val csName = rdr.getEncoding
-    rdr.close()
-    val source = Source.fromURL(uri.toURL, csName) // tbd: can one create a Source somehow directly from the rdr?
-    source
+
+    // must be buffered to support mark(), needed by heuristics
+    val is = new BufferedInputStream(uri.toURL.openStream())
+    val enc = EncodingHeuristics.readEncodingFromStream(is)
+    Source.fromInputStream(is, enc)
   }, true) {
 
   // This one line is a bit of a hack to get consistent line numbers. The