[PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle opened a new pull request, #1112:
URL: https://github.com/apache/daffodil/pull/1112

   Version 3.7.0-SNAPSHOT
   
   This was designed to support integration of Daffodil directly (without intermediate XML or JSON or even EXI) to other data handling tools, specifically, Apache Drill.
   
   InfosetElement and related InfosetNode traits are now in runtime1.api package.
   
   InfosetOutputter now has methods which use InfosetElement and InfosetArray traits as the objects passed to the handler methods. This improves the API over making DIArray, DISimple, and DIComplex visible.
   
   Added Metadata, ElementMetadata, etc. (also runtime1.api package) which provide limited exposure to the RuntimeData and CompilerInfo information.
   
   Added MetadataHandler - which is walked by MetadataWalker which can be called from DataProcessor.
   
   Walking the runtime1 metadata is easier than walking the DSOM tree. And these data fabrics like Apache Drill are interfacing to the runtime, not the compiler, so it's natural for the runtime1 metadata structures and data structures to be the ones driving the interfacing.
   
   The InfosetNode types were always supposed to be the API, the DINodes the implementation. This solves the issue of what classes should show through to SAPI and JAPI about the infoset nodes. It should be the InfosetNode types, not the DINode types.
   
   Furthermore, the InfosetNode types can have methods to access the needed runtime metadata information needed by walkers, InfosetOutputters, etc.
   
   These hide our infoset implementation and runtime metdata (RuntimeData classes) implementions.
   
   Added unit test for metadata and data walking to core module
   
   Note: Nothing has changed with InfosetInputters, as those are not needed for Apache Drill integration - which is parse-only.
   
   Refactored InfosetOutputter to enable sharing InfosetOutputterImpl.
   
   BlobMethodImpl is now a class so must be extended first, InfosetOutputter is a trait so is implemented (second) in Java declarations.
   
   Removed use of DIComplex, DISimple from SAPI and JAPI tests
   
   NodeInfo.PrimType is n.g. for Java access.
   Using PrimTypeNode which is a class.
   
   TODO: maybe just use type name strings for this.
   
   Added features to SchemaUtils to avoid "tns" prefix definition (which is now officially frowned upon)
   
   Added isHidden to SequenceRuntimeData. Needed to avoid walking hidden elements that appear in metadata structures, but aren't relevant as they do not appear in InfosetOutputter events.
   
   TODO: ensure these are not shared between hidden/non-hidden groups. (PropEnv stuff)
   That is: isHidden on SequenceRuntimeData may be broken in the case where a sequence definition is shared both hidden and non-hidden.
   
   Added child access to InfosetComplexElement. This is used by tests.
   
   It is questionable whether this should be allowed, because we don't have fast O(1) hash-based access to all children, only to children referenced by expressions. This allows linear search for now.
   
   Added QName.toPrefix_localName method. Why allocate these on the fly if they're static compile time constants.
   
   DAFFODIL-2832


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on code in PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#discussion_r1392923557


##########
daffodil-core/src/test/scala/org/apache/daffodil/core/infoset/TestInfoset.scala:
##########
@@ -183,16 +191,18 @@ class TestInfoset1 {
     val (infoset: DIComplex, _, tunable) = testInfoset(testSchema, xmlInfoset)
     val Seq(w_erd) = infoset.erd.childERDs
     infoset.getChildArray(w_erd, tunable) match {
-      case arr: DIArray => {
+      case arr: InfosetArray => {
         assertEquals(2, arr.length)
-        var a = arr(1).asInstanceOf[InfosetSimpleElement]
-        assertEquals(w_erd, a.runtimeData)
+        var a = arr(1).asInstanceOf[InfosetSimpleElement] // 1-based
+        assertEquals(w_erd, a.metadata)
 
-        assertEquals(4, a.dataValue.getAnyRef)
-        assertEquals(infoset, a.parent)
+        assertEquals(4, a.getUnsignedLong.longValue().toInt)

Review Comment:
   These return java types. java.math.BigInteger didn't seem to have a direct conversion to int. Hence this. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on code in PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#discussion_r1387444278


##########
daffodil-core/src/test/scala/org/apache/daffodil/core/util/TestUtils.scala:
##########
@@ -40,13 +40,13 @@ import org.apache.daffodil.lib.xml._
 import org.apache.daffodil.runtime1.api.DFDL
 import org.apache.daffodil.runtime1.debugger._
 import org.apache.daffodil.runtime1.infoset.InfosetInputter
-import org.apache.daffodil.runtime1.infoset.InfosetOutputter
 import org.apache.daffodil.runtime1.infoset.ScalaXMLInfosetInputter
 import org.apache.daffodil.runtime1.infoset.ScalaXMLInfosetOutputter
 import org.apache.daffodil.runtime1.processors.DataProcessor
 import org.apache.daffodil.runtime1.processors.VariableMap
 
 import org.apache.commons.io.output.NullOutputStream
+import org.apache.daffodil.runtime1.infoset.InfosetOutputter
 import org.junit.Assert.assertEquals
 

Review Comment:
   Revert. Just a re-ordering of imports. 



##########
daffodil-core/src/test/scala/org/apache/daffodil/core/general/TestRuntimeProperties.scala:
##########
@@ -37,7 +37,8 @@ import org.junit.Test
  */
 class RedactingScalaXMLInfosetOutputter extends ScalaXMLInfosetOutputter {
 
-  override def startSimple(diSimple: DISimple): Unit = {
+  override def startSimple(se: InfosetSimpleElement): Unit = {
+    val diSimple = se.asInstanceOf[DISimple]

Review Comment:
   This should be able to be done without need for DISimple.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetImpl.scala:
##########
@@ -14,13 +14,18 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-
 package org.apache.daffodil.runtime1.infoset
 
-import java.lang.{ Boolean => JBoolean }
-import java.lang.{ Number => JNumber }
-import java.math.{ BigDecimal => JBigDecimal }
-import java.util.HashMap
+import java.lang.{

Review Comment:
   Revise these imports into standard style.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetOutputter.scala:
##########
@@ -1,22 +1,9 @@
-/*

Review Comment:
   Restore banner.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,80 @@
+package org.apache.daffodil.runtime1.api
+
+import org.apache.daffodil.lib.exceptions.SchemaFileLocation
+import org.apache.daffodil.lib.xml.NS
+import org.apache.daffodil.lib.xml.NamedQName
+import org.apache.daffodil.runtime1.dpath.PrimTypeNode
+
+/*
+ * This is the supportable API for access to the RuntimeData structures
+ */
+trait Metadata {
+  def schemaFileLocation: SchemaFileLocation
+  def path: String
+  def diagnosticDebugName: String
+  override def toString = diagnosticDebugName
+}
+
+trait TermMetadata extends Metadata {
+  def isArray: Boolean
+}
+
+trait ElementMetadata extends TermMetadata {
+  def isSimpleType: Boolean
+  def isComplexType: Boolean
+
+  def name: String
+  def namespace: NS
+  def optNamespacePrefix: Option[String]
+  def isArray: Boolean
+  def isOptional: Boolean
+  def namedQName: NamedQName
+
+  def sscd: String
+  def isNillable: Boolean
+
+  def runtimeProperties: java.util.Map[String, String]
+
+}
+
+trait ComplexElementMetadata extends ElementMetadata {
+
+  def childMetadata: Seq[ElementMetadata]
+}
+
+trait SimpleElementMetadata extends ElementMetadata {
+  def primType: PrimTypeNode
+}
+
+trait ModelGroupMetadata extends TermMetadata {}
+
+trait SequenceMetadata extends ModelGroupMetadata {}
+
+trait ChoiceMetadata extends ModelGroupMetadata {}
+
+/**
+ * Base class used by clients who want to walk the runtime schema information.
+ */
+abstract class MetadataHandler() {
+
+  /**
+   * Called for simple type element metadata (for declarations or references)
+   */
+  def simpleElementMetadata(m: SimpleElementMetadata): Unit
+
+  /**
+   * Called for complex type element metadata (for declarations or references)
+   *
+   * Subsequent calls will be for the model group making up the content
+   * of the element.
+   */
+  def startComplexElementMetadata(m: ComplexElementMetadata): Unit
+  def endComplexElementMetadata(m: ComplexElementMetadata): Unit
+
+  //  def startSequenceMetadata(m: SequenceMetadata): Unit

Review Comment:
   Decide if these exist or not. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on code in PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#discussion_r1387457854


##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/MetadataWalker.scala:
##########
@@ -0,0 +1,76 @@
+package org.apache.daffodil.runtime1.processors

Review Comment:
   Apache banner missing



##########
daffodil-japi/src/main/scala/org/apache/daffodil/japi/Daffodil.scala:
##########
@@ -514,6 +508,13 @@ class DataProcessor private[japi] (private var dp: SDataProcessor)
    */
   def save(output: WritableByteChannel): Unit = dp.save(output)
 
+  /**
+   * Walks the handler over the runtime metadata structures
+   *
+   * @param handler - the handler is called-back during the walk as each metadata structure is encountered.
+   */
+  def walkMetadata(handler: MetadataHandler) = dp.walkMetadata(handler)

Review Comment:
   Comment from prior PR: 
   https://github.com/apache/daffodil/pull/1092/files#r1360829216
   
   Substance of it is why do we need another walker. Answer is in the commit message, but in short - because it makes more sense to do it on the runtime1 metadata objects since we're interfacing to runtime1, not to the schema compiler. 
   
   This may need more study. The NiFi integration uses a DSOM walker. Consolidation may be possible. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/MetadataWalker.scala:
##########
@@ -0,0 +1,76 @@
+package org.apache.daffodil.runtime1.processors

Review Comment:
   Comment from prior PR:
   https://github.com/apache/daffodil/pull/1092/files#r1360834018
   
   Asked if this should be in some specific different package like perhaps "dfdlx.japi" or japix ?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/MetadataWalker.scala:
##########
@@ -0,0 +1,76 @@
+package org.apache.daffodil.runtime1.processors
+
+import org.apache.daffodil.lib.exceptions.Assert
+import org.apache.daffodil.runtime1.api.MetadataHandler
+import org.apache.daffodil.runtime1.api.SequenceMetadata
+
+/**
+ * Walks the schema, but not the DSOM schema, it walks the RuntimeData objects that
+ * represent the DFDL schema at runtime.
+ *
+ * @param dp
+ */
+class MetadataWalker(private val dp: DataProcessor) {
+
+  private lazy val rootERD = dp.ssrd.elementRuntimeData
+
+  def walk(handler: MetadataHandler): Unit = {
+    walkTerm(handler, rootERD)
+  }
+
+  private def walkTerm(handler: MetadataHandler, trd: TermRuntimeData): Unit = {
+    trd match {
+      case err: ErrorERD => Assert.invariantFailed("should not get ErrorERDs")
+      case erd: ElementRuntimeData => walkElement(handler, erd)
+      case srd: SequenceRuntimeData => walkSequence(handler, srd)
+      case crd: ChoiceRuntimeData => walkChoice(handler, crd)
+      case _ => Assert.invariantFailed(s"unrecognized TermRuntimeData subtype: $trd")
+    }
+  }
+
+  private def walkElement(handler: MetadataHandler, erd: ElementRuntimeData): Unit = {
+    if (erd.optComplexTypeModelGroupRuntimeData.isDefined)
+      walkComplexElement(handler, erd)
+    else
+      walkSimpleElement(handler, erd)
+  }
+
+  private def walkComplexElement(
+    handler: MetadataHandler,
+    erd: ElementRuntimeData,
+  ): Unit = {
+    val mgrd = erd.optComplexTypeModelGroupRuntimeData.getOrElse {
+      Assert.invariantFailed("not a complex type element")
+    }
+    handler.startComplexElementMetadata(erd)
+    walkTerm(handler, mgrd)
+    handler.endComplexElementMetadata(erd)
+  }
+
+  private def walkSimpleElement(
+    handler: MetadataHandler,
+    erd: ElementRuntimeData,
+  ): Unit = {
+    handler.simpleElementMetadata(erd)
+  }
+
+  private def walkSequence(handler: MetadataHandler, sm: SequenceMetadata): Unit = {
+    val srd = sm.asInstanceOf[SequenceRuntimeData]
+    if (!srd.isHidden) {

Review Comment:
   Prior PR comment:
   https://github.com/apache/daffodil/pull/1092/files#r1360846271
   
   Was observing the need for this isHidden handling. 
   
   The commit comment has a TODO about verifying that we don't share groups between hidden/not. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["stevedlawrence (via GitHub)" <gi...@apache.org>]

stevedlawrence commented on code in PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#discussion_r1391117119


##########
daffodil-core/src/test/scala/org/apache/daffodil/core/api/TestMetadataWalking.scala:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.core.api
+
+import scala.collection.mutable.ArrayBuffer
+import scala.xml.Elem
+
+import org.apache.daffodil.core.util.TestUtils
+import org.apache.daffodil.io.InputSourceDataInputStream
+import org.apache.daffodil.lib.util._
+import org.apache.daffodil.runtime1.api.ChoiceMetadata
+import org.apache.daffodil.runtime1.api.ComplexElementMetadata
+import org.apache.daffodil.runtime1.api.DFDL.ParseResult
+import org.apache.daffodil.runtime1.api.ElementMetadata
+import org.apache.daffodil.runtime1.api.InfosetArray
+import org.apache.daffodil.runtime1.api.InfosetComplexElement
+import org.apache.daffodil.runtime1.api.InfosetElement
+import org.apache.daffodil.runtime1.api.InfosetItem
+import org.apache.daffodil.runtime1.api.InfosetSimpleElement
+import org.apache.daffodil.runtime1.api.Metadata
+import org.apache.daffodil.runtime1.api.MetadataHandler
+import org.apache.daffodil.runtime1.api.SequenceMetadata
+import org.apache.daffodil.runtime1.api.SimpleElementMetadata
+import org.apache.daffodil.runtime1.infoset.ExceptionOccurredMixin
+import org.apache.daffodil.runtime1.infoset.InfosetOutputter
+import org.apache.daffodil.runtime1.infoset.InfosetOutputterImpl
+import org.apache.daffodil.runtime1.processors.DataProcessor
+
+import org.junit.Assert.assertEquals
+import org.junit.Assert.assertTrue
+import org.junit.Test
+
+class TestMetadataWalking {
+
+  def compileAndWalkMetadata(schema: Elem, mh: MetadataHandler): DataProcessor = {
+    val dp = TestUtils.compileSchema(schema)
+    assertTrue(!dp.isError)
+    dp.walkMetadata(mh)
+    dp
+  }
+
+  def parseAndWalkData(dp: DataProcessor, infosetOutputter: InfosetOutputter)(
+    data: Array[Byte],
+  ): ParseResult = {
+    val isdis = InputSourceDataInputStream(data)
+    val res = dp.parse(isdis, infosetOutputter)
+    res
+  }
+
+  class GatherMetadata extends MetadataHandler {
+
+    private val buf = new ArrayBuffer[Metadata]();
+
+    def getResult: Seq[Metadata] = {
+      val res: Seq[Metadata] = buf.toVector // makes a copy
+      buf.clear()
+      res
+    }
+
+    override def simpleElementMetadata(m: SimpleElementMetadata): Unit = buf += m
+
+    override def startComplexElementMetadata(m: ComplexElementMetadata): Unit = buf += m
+
+    override def endComplexElementMetadata(m: ComplexElementMetadata): Unit = buf += m
+
+    override def startSequenceMetadata(m: SequenceMetadata): Unit = buf += m
+
+    override def endSequenceMetadata(m: SequenceMetadata): Unit = buf += m
+
+    override def startChoiceMetadata(m: ChoiceMetadata): Unit = buf += m
+
+    override def endChoiceMetadata(m: ChoiceMetadata): Unit = buf += m
+  }
+
+  class GatherData
+    extends InfosetOutputterImpl
+    with InfosetOutputter
+    with ExceptionOccurredMixin {
+
+    private val buf = new ArrayBuffer[InfosetItem]
+
+    def getResult: Seq[InfosetItem] = {
+      val res = buf.toVector
+      reset()
+      res
+    }
+
+    override def reset(): Unit = { buf.clear() }
+
+    override def startDocument(): Unit = {}
+
+    override def endDocument(): Unit = {}
+
+    override def startSimple(diSimple: InfosetSimpleElement): Unit = { buf += diSimple }
+
+    override def endSimple(diSimple: InfosetSimpleElement): Unit = {}

Review Comment:
   This is slightly different for the MetaDataHandler, which just has a "simpleElement" function, instead of separate start/endSimple functions. Having start/end events for simple in the InfosetOutputter API maybe have been a mistake. In fact, most implementations make endSimple be a no-op. The only one that doesn't is the saxInfosetOutputter, which could do all it's logic in startSimple.
   
   We are already breaking backwards compatibility pretty heavily here, maybe we should also drop endSimple and just have a single method for handling simple elements in the InfosetOutputter?



##########
daffodil-japi/src/test/java/org/apache/daffodil/example/TestInfosetOutputter.java:
##########
@@ -51,45 +50,50 @@ public void endDocument() {
     }
 
     @Override
-    public void startSimple(DISimple diSimple) {
+    public void startSimple(InfosetSimpleElement diSimple) {
         events.add(
             TestInfosetEvent.startSimple(
-                diSimple.erd().name(),
-                diSimple.erd().namedQName().namespace().toString(),
+                diSimple.metadata().name(),
+                diSimple.metadata().namespace(),
                 diSimple.dataValueAsString(),
-                diSimple.erd().isNillable() ? diSimple.isNilled() : null));
+                diSimple.metadata().isNillable() ? diSimple.isNilled() : null));

Review Comment:
   Rename `diSimple` to just `simple`? Same with other tests.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.api
+
+import scala.xml.NamespaceBinding
+
+/**
+ * This is the supportable API for access to the RuntimeData structures
+ * which provide access to static information about a given .
+ *
+ * This is used to interface other data processing fabrics to Daffodil
+ * data and metadata, by mapping to/from these metadata objects.
+ */
+trait Metadata {
+
+  /**
+   * Provides the file context of a metadata component. This refers to the specific
+   * DFDL schema file where the corresponding DFDL schema text resides corresponding
+   * to this metadata object.
+   * <p/>
+   * This is for use in diagnostic messaging. It is not the actual file URI, because
+   * those may contain personal-identifying information about the person/acccount and
+   * system that compiled the schema. It will provide enough content about the file URI that
+   * a user will be able to identify which file, but some prefix of the path
+   * components trimmed to make it of a manageable length.
+   * <p/>
+   * Used along with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]]
+   * and [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineColumnNumber]],
+   * this can give a precise location in the DFDL schema file.
+   * @return a string containing the file information, or null if unknown.
+   */
+  def schemaFileInfo: String
+
+  /**
+   * Provides the line number to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileInfo]].
+   * @return the line number as a string, or null if unknown.
+   */
+  def schemaFileLineNumber: String
+
+  /**
+   * Provides the column number within the text line, to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]].
+   * @return the column number within the text line, as a string, or null if unknown.
+   */
+  def schemaFileLineColumnNumber: String

Review Comment:
   Can line/column be Long's instead? Or does Daffodil internally represent these as strings so it's just easier to pass that through? If we need the `null` to mean for no-information, they can be boxed java longs so we can still return null?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetOutputter.scala:
##########
@@ -84,75 +85,98 @@ trait InfosetOutputter {
    *                 value, nil, name, namespace, etc.
    */
   @throws[Exception]
-  def endSimple(diSimple: DISimple): Unit
+  def endSimple(diSimple: InfosetSimpleElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of a complex element.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the complex element that is started. Various fields of
+   * @param complex the complex element that is started. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    */
   @throws[Exception]
-  def startComplex(diComplex: DIComplex): Unit
+  def startComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the end of a complex element.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the complex element that is ended. Various fields of
+   * @param complex the complex element that is ended. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    */
   @throws[Exception]
-  def endComplex(diComplex: DIComplex): Unit
+  def endComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of an array of elements.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the array that is started. Various fields of
+   * @param array the array that is started. Various fields of
    *                  DIArray can be accessed to determine things like the
    *                  name, namespace, etc.
    */
   @throws[Exception]
-  def startArray(diArray: DIArray): Unit
+  def startArray(array: InfosetArray): Unit
 
   /**
    * Called by Daffodil internals to signify the end of an array of elements.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the array that is ended. Various fields of
+   * @param array the array that is ended. Various fields of
    *                  DIArray can be accessed to determine things like the
    *                  name, namespace, etc.
    */
   @throws[Exception]
-  def endArray(diArray: DIArray): Unit
+  def endArray(array: InfosetArray): Unit
 
-  def getStatus(): Status = {
-    // Done, Ready (Not started), Visiting (part way done - can retry to visit more)...
-    status
-  }
+  def getStatus(): Status
 
   /**
-   * Helper function to determine if an element is nilled or not, taking into
-   * account whether or not the nilled state has been set yet.
+   * Override this to be fed any exceptions thrown by the start/end methods of this class.
+   * This is mostly to facilitate users debugging their code that uses the API.
    *
-   * @param diElement the element to check the nilled state of
+   * This saves the user of the API from having to wrap try/catch around their
+   * start/end method bodies.
    *
-   * @return true if the nilled state has been set and is true. false if the
-   *         nilled state is false or if the nilled state has not been set yet
-   *         (e.g. during debugging)
+   * @param e the exception thrown by one of the other methods.
    */
-  final def isNilled(diElement: DIElement): Boolean = {
-    val maybeIsNilled = diElement.maybeIsNilled
-    maybeIsNilled.isDefined && maybeIsNilled.get == true
-  }
+  def exceptionOccurred(e: Exception): Unit
+
+}
+
+/**
+ * Methods that provide Blob (Binary Large Object) support.
+ *
+ * FIXME: Scaladoc
+ */
+trait BlobMethodsMixin {
+
+  def setBlobAttributes(blobDir: Path, blobPrefix: String, blobSuffix: String): Unit
+  def setBlobPaths(empty: Seq[Path]): Unit
+  def getBlobDirectory(): Path
+  def getBlobPrefix(): String
+  def getBlobSuffix(): String
+
+}
+
+/**
+ * An available basic implementation of the BLOB methods.
+ * Stores blobs in files in directory identified by Java system property
+ * `java.io.tempdir`.
+ *
+ * FIXME: Unclear if Daffodil implements blob copying based on these
+ *   alone or not, and what else people have to do for BLOB support
+ *   than mixin this to their InfosetOutputter. 

Review Comment:
   Nothing needs to be done to support blobs. These functions are just a way to configure where blobs get written to. That information is set on the infoset outputter because there isn't really a good other place to do it. This also provides a way to get paths to all blobs wihtout having to traverse the infoset.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/JsonInfosetOutputter.scala:
##########
@@ -112,11 +116,12 @@ class JsonInfosetOutputter private (writer: java.io.Writer, pretty: Boolean)
     }
   }
 
-  override def endSimple(simple: DISimple): Unit = {
+  override def endSimple(se: InfosetSimpleElement): Unit = {
     // nothing to do
   }
 
-  override def startComplex(complex: DIComplex): Unit = {
+  override def startComplex(ce: InfosetComplexElement): Unit = {
+    val complex = ce.asInstanceOf[DIComplex]

Review Comment:
   Can this asInstanceOf go away? Same with others in the class. Or is that planned for a later PR?



##########
daffodil-core/src/test/scala/org/apache/daffodil/core/api/TestMetadataWalking.scala:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.core.api
+
+import scala.collection.mutable.ArrayBuffer
+import scala.xml.Elem
+
+import org.apache.daffodil.core.util.TestUtils
+import org.apache.daffodil.io.InputSourceDataInputStream
+import org.apache.daffodil.lib.util._
+import org.apache.daffodil.runtime1.api.ChoiceMetadata
+import org.apache.daffodil.runtime1.api.ComplexElementMetadata
+import org.apache.daffodil.runtime1.api.DFDL.ParseResult
+import org.apache.daffodil.runtime1.api.ElementMetadata
+import org.apache.daffodil.runtime1.api.InfosetArray
+import org.apache.daffodil.runtime1.api.InfosetComplexElement
+import org.apache.daffodil.runtime1.api.InfosetElement
+import org.apache.daffodil.runtime1.api.InfosetItem
+import org.apache.daffodil.runtime1.api.InfosetSimpleElement
+import org.apache.daffodil.runtime1.api.Metadata
+import org.apache.daffodil.runtime1.api.MetadataHandler
+import org.apache.daffodil.runtime1.api.SequenceMetadata
+import org.apache.daffodil.runtime1.api.SimpleElementMetadata
+import org.apache.daffodil.runtime1.infoset.ExceptionOccurredMixin
+import org.apache.daffodil.runtime1.infoset.InfosetOutputter
+import org.apache.daffodil.runtime1.infoset.InfosetOutputterImpl
+import org.apache.daffodil.runtime1.processors.DataProcessor
+
+import org.junit.Assert.assertEquals
+import org.junit.Assert.assertTrue
+import org.junit.Test
+
+class TestMetadataWalking {
+
+  def compileAndWalkMetadata(schema: Elem, mh: MetadataHandler): DataProcessor = {
+    val dp = TestUtils.compileSchema(schema)
+    assertTrue(!dp.isError)
+    dp.walkMetadata(mh)
+    dp
+  }
+
+  def parseAndWalkData(dp: DataProcessor, infosetOutputter: InfosetOutputter)(
+    data: Array[Byte],
+  ): ParseResult = {
+    val isdis = InputSourceDataInputStream(data)
+    val res = dp.parse(isdis, infosetOutputter)
+    res
+  }
+
+  class GatherMetadata extends MetadataHandler {
+
+    private val buf = new ArrayBuffer[Metadata]();
+
+    def getResult: Seq[Metadata] = {
+      val res: Seq[Metadata] = buf.toVector // makes a copy
+      buf.clear()
+      res
+    }
+
+    override def simpleElementMetadata(m: SimpleElementMetadata): Unit = buf += m
+
+    override def startComplexElementMetadata(m: ComplexElementMetadata): Unit = buf += m
+
+    override def endComplexElementMetadata(m: ComplexElementMetadata): Unit = buf += m
+
+    override def startSequenceMetadata(m: SequenceMetadata): Unit = buf += m
+
+    override def endSequenceMetadata(m: SequenceMetadata): Unit = buf += m
+
+    override def startChoiceMetadata(m: ChoiceMetadata): Unit = buf += m
+
+    override def endChoiceMetadata(m: ChoiceMetadata): Unit = buf += m
+  }
+
+  class GatherData
+    extends InfosetOutputterImpl
+    with InfosetOutputter
+    with ExceptionOccurredMixin {
+
+    private val buf = new ArrayBuffer[InfosetItem]
+
+    def getResult: Seq[InfosetItem] = {
+      val res = buf.toVector
+      reset()
+      res
+    }
+
+    override def reset(): Unit = { buf.clear() }
+
+    override def startDocument(): Unit = {}
+
+    override def endDocument(): Unit = {}

Review Comment:
   Does a MetadatHandler not  of a start/endDocument like an InfosetOutputter does?



##########
daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/SequenceTermRuntime1Mixin.scala:
##########
@@ -82,6 +83,7 @@ trait ChoiceBranchImpliedSequenceRuntime1Mixin { self: ChoiceBranchImpliedSequen
       FillByteUseNotAllowedEv,
       Maybe.Nope,
       Maybe.Nope,
+      false,

Review Comment:
   Suggest `isHidden = false` like in other places?



##########
daffodil-core/src/test/scala/org/apache/daffodil/core/api/TestMetadataWalking.scala:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.core.api
+
+import scala.collection.mutable.ArrayBuffer
+import scala.xml.Elem
+
+import org.apache.daffodil.core.util.TestUtils
+import org.apache.daffodil.io.InputSourceDataInputStream
+import org.apache.daffodil.lib.util._
+import org.apache.daffodil.runtime1.api.ChoiceMetadata
+import org.apache.daffodil.runtime1.api.ComplexElementMetadata
+import org.apache.daffodil.runtime1.api.DFDL.ParseResult
+import org.apache.daffodil.runtime1.api.ElementMetadata
+import org.apache.daffodil.runtime1.api.InfosetArray
+import org.apache.daffodil.runtime1.api.InfosetComplexElement
+import org.apache.daffodil.runtime1.api.InfosetElement
+import org.apache.daffodil.runtime1.api.InfosetItem
+import org.apache.daffodil.runtime1.api.InfosetSimpleElement
+import org.apache.daffodil.runtime1.api.Metadata
+import org.apache.daffodil.runtime1.api.MetadataHandler
+import org.apache.daffodil.runtime1.api.SequenceMetadata
+import org.apache.daffodil.runtime1.api.SimpleElementMetadata
+import org.apache.daffodil.runtime1.infoset.ExceptionOccurredMixin
+import org.apache.daffodil.runtime1.infoset.InfosetOutputter
+import org.apache.daffodil.runtime1.infoset.InfosetOutputterImpl
+import org.apache.daffodil.runtime1.processors.DataProcessor
+
+import org.junit.Assert.assertEquals
+import org.junit.Assert.assertTrue
+import org.junit.Test
+
+class TestMetadataWalking {
+
+  def compileAndWalkMetadata(schema: Elem, mh: MetadataHandler): DataProcessor = {
+    val dp = TestUtils.compileSchema(schema)
+    assertTrue(!dp.isError)
+    dp.walkMetadata(mh)
+    dp
+  }
+
+  def parseAndWalkData(dp: DataProcessor, infosetOutputter: InfosetOutputter)(
+    data: Array[Byte],
+  ): ParseResult = {
+    val isdis = InputSourceDataInputStream(data)
+    val res = dp.parse(isdis, infosetOutputter)
+    res
+  }
+
+  class GatherMetadata extends MetadataHandler {
+
+    private val buf = new ArrayBuffer[Metadata]();
+
+    def getResult: Seq[Metadata] = {
+      val res: Seq[Metadata] = buf.toVector // makes a copy
+      buf.clear()
+      res
+    }
+
+    override def simpleElementMetadata(m: SimpleElementMetadata): Unit = buf += m
+
+    override def startComplexElementMetadata(m: ComplexElementMetadata): Unit = buf += m
+
+    override def endComplexElementMetadata(m: ComplexElementMetadata): Unit = buf += m
+
+    override def startSequenceMetadata(m: SequenceMetadata): Unit = buf += m
+
+    override def endSequenceMetadata(m: SequenceMetadata): Unit = buf += m
+
+    override def startChoiceMetadata(m: ChoiceMetadata): Unit = buf += m
+
+    override def endChoiceMetadata(m: ChoiceMetadata): Unit = buf += m

Review Comment:
   Do we need `Metadata` in all the names of these functions? Seems unnecessarily verbose since we're already in a Metaata handler?



##########
daffodil-japi/src/main/scala/org/apache/daffodil/japi/Daffodil.scala:
##########
@@ -514,6 +515,20 @@ class DataProcessor private[japi] (private var dp: SDataProcessor)
    */
   def save(output: WritableByteChannel): Unit = dp.save(output)
 
+  /**
+   * Walks the handler over the runtime [[org.apache.daffodil.runtime1.api.Metadata]] structures.
+   * These provide information about name, namespace, type, simple/complex, etc.
+   *
+   * This is used to interface Daffodil runtime1 metadata to the metadata structures
+   * of other software systems.
+   *
+   * See [[org.apache.daffodil.runtime1.api.MetadataHandler]] for more motivating materials about
+   * runtime1 metadata walking.
+   *
+   * @param handler - the handler is called-back during the walk as each metadata structure is encountered.
+   */
+  def walkMetadata(handler: MetadataHandler): Unit = dp.walkMetadata(handler)

Review Comment:
   We should implement tests in TestJavaAPI.java to get coverage from this and make sure everything is easily accessible using reasonable Java without any weird scala-isms.



##########
daffodil-japi/src/main/scala/org/apache/daffodil/japi/infoset/Infoset.scala:
##########
@@ -154,69 +153,69 @@ abstract class InfosetOutputter extends SInfosetOutputter {
   /**
    * Called by Daffodil internals to signify the beginning of a simple element.
    *
-   * @param diSimple the simple element that is started. Various fields of
+   * @param simple the simple element that is started. Various fields of
    *                 DISimple can be accessed to determine things like the
    *                 value, nil, name, namespace, etc.
    * @throws Exception if there was an error and Daffodil should stop parsing
    */
 
   @throws[Exception]
-  def startSimple(diSimple: DISimple): Unit
+  def startSimple(simple: InfosetSimpleElement): Unit

Review Comment:
   When we look at java doc, we need to make sure to see what this looks like. InfosetSimpleElement and the others are not in the daffodil-japi project, so I dont' think will have any java doc associated with them. Maybe we should consider options to make that available? I really don't like our the proxy kinof thing we've been doing in past, I'm not sure if there are other alternatives though.



##########
daffodil-japi/src/main/scala/org/apache/daffodil/japi/infoset/Infoset.scala:
##########
@@ -154,69 +153,69 @@ abstract class InfosetOutputter extends SInfosetOutputter {
   /**
    * Called by Daffodil internals to signify the beginning of a simple element.
    *
-   * @param diSimple the simple element that is started. Various fields of
+   * @param simple the simple element that is started. Various fields of
    *                 DISimple can be accessed to determine things like the
    *                 value, nil, name, namespace, etc.
    * @throws Exception if there was an error and Daffodil should stop parsing
    */
 
   @throws[Exception]
-  def startSimple(diSimple: DISimple): Unit
+  def startSimple(simple: InfosetSimpleElement): Unit
 
   /**
    * Called by Daffodil internals to signify the end of a simple element.
    *
-   * @param diSimple the simple element that is ended. Various fields of
+   * @param simple the simple element that is ended. Various fields of
    *                 DISimple can be accessed to determine things like the
    *                 value, nil, name, namespace, etc.
    * @throws Exception if there was an error and Daffodil should stop parsing
    */
   @throws[Exception]
-  def endSimple(diSimple: DISimple): Unit
+  def endSimple(simple: InfosetSimpleElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of a complex element.
    *
-   * @param diComplex the complex element that is started. Various fields of
+   * @param complex the complex element that is started. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    * @throws Exception if there was an error and Daffodil should stop parsing
    */
   @throws[Exception]
-  def startComplex(diComplex: DIComplex): Unit
+  def startComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the end of a complex element.
    *
-   * @param diComplex the complex element that is ended. Various fields of
+   * @param complex the complex element that is ended. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    * @throws Exception if there was an error and Daffodil should stop parsing
    */
   @throws[Exception]
-  def endComplex(diComplex: DIComplex): Unit
+  def endComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of an array of elements.
    *
-   * @param diArray the array that is started. Various fields of
+   * @param array the array that is started. Various fields of
    *                DIArray can be accessed to determine things like the
    *                name, namespace, etc.
    * @throws Exception if there was an error and Daffodil should stop parsing
    */
   @throws[Exception]
-  def startArray(diArray: DIArray): Unit
+  def startArray(array: InfosetArray): Unit
 
   /**
    * Called by Daffodil internals to signify the end of an array of elements.
    *
-   * @param diArray the array that is ended. Various fields of
+   * @param array the array that is ended. Various fields of
    *                DIArray can be accessed to determine things like the
    *                name, namespace, etc.
    * @throws Exception if there was an error and Daffodil should stop parsing
    */
   @throws[Exception]
-  def endArray(diArray: DIArray): Unit
+  def endArray(array: InfosetArray): Unit

Review Comment:
   This is all a pretty big backwards-icompabitle change. Maybe that's fine and will just be another thing that warrants the next version being 4.0.0.
   
   We could maybe do somthing where this API has both variants of functions, and the new one calls the old one, e.g.
   ```scala
   def endArray(diArray: DIArray): Unit
   def endArray(array: InfosetAray): Unit = endArray(array.asInstanceOf[DIArray])
   ```
   This way old infoset outputters still work, but new infoset ouputters can be written with the new API. Not sure the extra complication is worth it, especially if we go to 4.0.0.
   
   We should add a `Deprecation/Compatibility` section to the commit message though so we remember to mention this in the release notes.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import com.ibm.icu.util.Calendar
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+
+  /**
+   * @return the metadata of the element that is an array
+   */
+  override def metadata: ElementMetadata
+
+  /**
+   * @return the number of items in the array.
+   */
+  def length: Long
+
+  /**
+   * Access to an infoset element of the array.
+   * @param occursIndex1b the one-based index of the element to be accessed
+   * @return the infoset element located at the given index
+   */
+  def apply(occursIndex1b: Long): InfosetElement
+}
+
+/**
+ * API access to elements of the DFDL Infoset of both
+ * complex and simple type.
+ */
+trait InfosetElement extends InfosetItem {
+
+  /**
+   * In DFDL both simple and complex type elements can be
+   * nilled.
+   *
+   * @return true if the element is nilled, false otherwise.
+   */
+  def isNilled: Boolean
+
+  /**
+   * @return true if the element is a string or hexBinary of length 0, false otherwise.

Review Comment:
   Seems an odd special case to support. Can't the user just get the string or byte array and ask if that is empty? What are the cases where users would even care if they are empty? For example, empty string often isn't a special case.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import com.ibm.icu.util.Calendar
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+
+  /**
+   * @return the metadata of the element that is an array
+   */
+  override def metadata: ElementMetadata
+
+  /**
+   * @return the number of items in the array.
+   */
+  def length: Long
+
+  /**
+   * Access to an infoset element of the array.
+   * @param occursIndex1b the one-based index of the element to be accessed
+   * @return the infoset element located at the given index
+   */
+  def apply(occursIndex1b: Long): InfosetElement

Review Comment:
   What are array `length` add `apply` used for? My concern is when an `InfosetWalker` walks an array, the full length might not be known at that time. I don't remember if we wait for an array to be completely finished before walking into it (I would guess we don't?), but if we don't this could cause weird behavior. Can we get away with removing these?



##########
daffodil-japi/src/test/java/org/apache/daffodil/example/TestInfosetOutputter.java:
##########
@@ -51,45 +50,50 @@ public void endDocument() {
     }
 
     @Override
-    public void startSimple(DISimple diSimple) {
+    public void startSimple(InfosetSimpleElement diSimple) {
         events.add(
             TestInfosetEvent.startSimple(
-                diSimple.erd().name(),
-                diSimple.erd().namedQName().namespace().toString(),
+                diSimple.metadata().name(),
+                diSimple.metadata().namespace(),
                 diSimple.dataValueAsString(),
-                diSimple.erd().isNillable() ? diSimple.isNilled() : null));
+                diSimple.metadata().isNillable() ? diSimple.isNilled() : null));
     }
 
     @Override
-    public void endSimple(DISimple diSimple) {
+    public void endSimple(InfosetSimpleElement diSimple) {
         events.add(
             TestInfosetEvent.endSimple(
-                diSimple.erd().name(),
-                diSimple.erd().namedQName().namespace().toString()));
+                diSimple.metadata().name(),
+                diSimple.metadata().namespace()));
     }
 
     @Override
-    public void startComplex(DIComplex diComplex) {
+    public void startComplex(InfosetComplexElement complex) {
         events.add(
             TestInfosetEvent.startComplex(
-                diComplex.erd().name(),
-                diComplex.erd().namedQName().namespace().toString(),
-                diComplex.erd().isNillable() ? diComplex.isNilled() : null));
+                complex.metadata().name(),
+                complex.metadata().namespace(),
+                complex.metadata().isNillable() ? complex.isNilled() : null));

Review Comment:
   Unrlated to this PR, but do you know if we must call `isNillabled` before asking `isNilled`? Seems odd if so. It would be nice if isNilled just returns false if something isn't nillable. But maybe this test infoset outputter just doing something fancy and wants differentiable nillable but not nil vs not nillable.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import com.ibm.icu.util.Calendar
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+
+  /**
+   * @return the metadata of the element that is an array
+   */
+  override def metadata: ElementMetadata
+
+  /**
+   * @return the number of items in the array.
+   */
+  def length: Long
+
+  /**
+   * Access to an infoset element of the array.
+   * @param occursIndex1b the one-based index of the element to be accessed
+   * @return the infoset element located at the given index
+   */
+  def apply(occursIndex1b: Long): InfosetElement
+}
+
+/**
+ * API access to elements of the DFDL Infoset of both
+ * complex and simple type.
+ */
+trait InfosetElement extends InfosetItem {
+
+  /**
+   * In DFDL both simple and complex type elements can be
+   * nilled.
+   *
+   * @return true if the element is nilled, false otherwise.
+   */
+  def isNilled: Boolean
+
+  /**
+   * @return true if the element is a string or hexBinary of length 0, false otherwise.
+   */
+  def isEmpty: Boolean
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[ElementMetadata]]
+   */
+  def metadata: ElementMetadata
+
+}
+
+/**
+ * Methods specific complex elements in the infoset
+ */
+trait InfosetComplexElement extends InfosetElement {
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[ComplexElementMetadata]]
+   */
+  override def metadata: ComplexElementMetadata
+}
+
+/**
+ * Methods specific to simple elements in the infoset
+ */
+trait InfosetSimpleElement extends InfosetElement {
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[SimpleElementMetadata]]
+   */
+  override def metadata: SimpleElementMetadata
+
+  /**
+   * @return true if the element has a value, false for a nilled element.
+   */
+  def hasValue: Boolean

Review Comment:
   How is this different from the `isNilled` method inherited from `InfosetElement`. Seems if isNilled is false, then the element must have a value. Can we drop this to simplify the API?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import com.ibm.icu.util.Calendar
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+
+  /**
+   * @return the metadata of the element that is an array
+   */
+  override def metadata: ElementMetadata
+
+  /**
+   * @return the number of items in the array.
+   */
+  def length: Long
+
+  /**
+   * Access to an infoset element of the array.
+   * @param occursIndex1b the one-based index of the element to be accessed
+   * @return the infoset element located at the given index
+   */
+  def apply(occursIndex1b: Long): InfosetElement
+}
+
+/**
+ * API access to elements of the DFDL Infoset of both
+ * complex and simple type.
+ */
+trait InfosetElement extends InfosetItem {
+
+  /**
+   * In DFDL both simple and complex type elements can be
+   * nilled.
+   *
+   * @return true if the element is nilled, false otherwise.
+   */
+  def isNilled: Boolean
+
+  /**
+   * @return true if the element is a string or hexBinary of length 0, false otherwise.
+   */
+  def isEmpty: Boolean
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[ElementMetadata]]
+   */
+  def metadata: ElementMetadata
+
+}
+
+/**
+ * Methods specific complex elements in the infoset
+ */
+trait InfosetComplexElement extends InfosetElement {
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[ComplexElementMetadata]]
+   */
+  override def metadata: ComplexElementMetadata
+}
+
+/**
+ * Methods specific to simple elements in the infoset
+ */
+trait InfosetSimpleElement extends InfosetElement {
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[SimpleElementMetadata]]
+   */
+  override def metadata: SimpleElementMetadata
+
+  /**
+   * @return true if the element has a value, false for a nilled element.
+   */
+  def hasValue: Boolean
+
+  /**
+   * Obtains the value, then converts it to a string.
+   * Caches the string so we're not allocating strings repeatedly
+   */
+  def dataValueAsString: String

Review Comment:
   Thoughts on something like `getText` instead? This follows the `getXYZ` convention of the other functions, but maybe makes it clear that it is getting a textual representation of the underlying value?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import com.ibm.icu.util.Calendar
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+
+  /**
+   * @return the metadata of the element that is an array
+   */
+  override def metadata: ElementMetadata
+
+  /**
+   * @return the number of items in the array.
+   */
+  def length: Long
+
+  /**
+   * Access to an infoset element of the array.
+   * @param occursIndex1b the one-based index of the element to be accessed
+   * @return the infoset element located at the given index
+   */
+  def apply(occursIndex1b: Long): InfosetElement
+}
+
+/**
+ * API access to elements of the DFDL Infoset of both
+ * complex and simple type.
+ */
+trait InfosetElement extends InfosetItem {
+
+  /**
+   * In DFDL both simple and complex type elements can be
+   * nilled.
+   *
+   * @return true if the element is nilled, false otherwise.
+   */
+  def isNilled: Boolean
+
+  /**
+   * @return true if the element is a string or hexBinary of length 0, false otherwise.
+   */
+  def isEmpty: Boolean
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[ElementMetadata]]
+   */
+  def metadata: ElementMetadata
+
+}
+
+/**
+ * Methods specific complex elements in the infoset
+ */
+trait InfosetComplexElement extends InfosetElement {
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[ComplexElementMetadata]]
+   */
+  override def metadata: ComplexElementMetadata
+}
+
+/**
+ * Methods specific to simple elements in the infoset
+ */
+trait InfosetSimpleElement extends InfosetElement {
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[SimpleElementMetadata]]
+   */
+  override def metadata: SimpleElementMetadata
+
+  /**
+   * @return true if the element has a value, false for a nilled element.
+   */
+  def hasValue: Boolean
+
+  /**
+   * Obtains the value, then converts it to a string.
+   * Caches the string so we're not allocating strings repeatedly
+   */
+  def dataValueAsString: String
+
+  /*
+   * These are so that API users don't have to know about our
+   * very Scala-oriented DataValue type system.
+   */
+
+  /**
+   * @return the value of this simple element as a Scala AnyRef, which is
+   *         equivalent to a Java Object.
+   */
+  def getAnyRef: AnyRef
+
+  /**
+   * @return the value of this simple element as an Object (java.lang.Object),
+   *         which is equivalent to Scala AnyRef.
+   */
+  final def getObject: java.lang.Object = getAnyRef
+
+  // Note: I could not get @throws in scaladoc to work right.
+  // Complains "Could not find any member to link for ... and I tried various formulations of
+  // ClassCastException, with package, with and without [[..]].
+  // So I've just converted it to plain text.
+
+  /**
+   * @return Casts the value of this simple element as a java.math.BigDecimal.
+   * <p/>throws `ClassCastException` if the element value is not of type Decimal.
+   */
+  def getBigDecimal: JBigDecimal
+
+  /**
+   * @return Casts the value of this Date, Time, or DateTime simple element as a
+   *         `com.ibm.icu.util.Calendar`
+   * <p/>throws `ClassCastException` if the element value is not of type Date, Time, or DateTime
+   */
+  def getCalendar: Calendar
+
+  /**
+   * @return the value of this simple element cast to `com.ibm.icu.util.Calendar`.
+   * <p/>throws `ClassCastException` if the element value is not of the required type.
+   */
+  def getDate: Calendar
+
+  /**
+   * @return the value of this simple element cast to `com.ibm.icu.util.Calendar`.
+   * <p/>throws `ClassCastException` if the element value is not of the required type.
+   */
+  def getTime: Calendar
+
+  /**
+   * @return the value of this simple element cast to `com.ibm.icu.util.Calendar`.
+   * <p/>throws `ClassCastException` if the element value is not of the required type.
+   */
+  def getDateTime: Calendar
+
+  /**
+   * @return the value of this simple element of HexBinary type cast to `Array[Byte]`.
+   * <p/>throws `ClassCastException` if the element value is not of HexBinary type.
+   */
+  def getByteArray: Array[Byte]
+
+  /**
+   * @return the value of this simple element of Boolean type cast to java.lang.Boolean.
+   * <p/>throws `ClassCastException` if the element value is not of Boolean type.
+   */
+  def getBoolean: JBoolean
+
+  /**
+   * @return the value of this simple element of numeric type cast to java.lang.Number.
+   * <p/>throws `ClassCastException` if the element value is not of numeric type.
+   */
+  def getNumber: JNumber

Review Comment:
   Suggest we remove getNumber? The only useful functions provided by Number ways to get the number as a byte, short, int, long, float, or double. Presumably users are going to need to know which type of number they need anyways and call the right getXZY function?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.api
+
+import scala.xml.NamespaceBinding
+
+/**
+ * This is the supportable API for access to the RuntimeData structures
+ * which provide access to static information about a given .
+ *
+ * This is used to interface other data processing fabrics to Daffodil
+ * data and metadata, by mapping to/from these metadata objects.
+ */
+trait Metadata {
+
+  /**
+   * Provides the file context of a metadata component. This refers to the specific
+   * DFDL schema file where the corresponding DFDL schema text resides corresponding
+   * to this metadata object.
+   * <p/>
+   * This is for use in diagnostic messaging. It is not the actual file URI, because
+   * those may contain personal-identifying information about the person/acccount and
+   * system that compiled the schema. It will provide enough content about the file URI that
+   * a user will be able to identify which file, but some prefix of the path
+   * components trimmed to make it of a manageable length.
+   * <p/>
+   * Used along with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]]
+   * and [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineColumnNumber]],
+   * this can give a precise location in the DFDL schema file.
+   * @return a string containing the file information, or null if unknown.
+   */
+  def schemaFileInfo: String
+
+  /**
+   * Provides the line number to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileInfo]].
+   * @return the line number as a string, or null if unknown.
+   */
+  def schemaFileLineNumber: String
+
+  /**
+   * Provides the column number within the text line, to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]].
+   * @return the column number within the text line, as a string, or null if unknown.
+   */
+  def schemaFileLineColumnNumber: String
+
+  /**
+   * The name of the schema component, in a form suitable for diagnostic messages.
+   * Unnamed components like sequence or choice groups have a diagnosticDebugName, despite not having
+   * any actual name.
+   * @return the name of the component, suitable for use in diagnostic messages.
+   */
+  def diagnosticDebugName: String
+
+  /**
+   * The standard conversion of the metadata component to a string.
+   * Unless overridden this uses the [[org.apache.daffodil.runtime1.api.Metadata.diagnosticDebugName]].
+   * @return string representation of this metadata component.
+   */
+  override def toString = diagnosticDebugName
+}
+
+/*
+ * Provides metadata access that is common to all Terms, which include
+ * Elements of simple or complex type, as well as the Sequence and Choice groups.
+ */
+trait TermMetadata extends Metadata {
+  // nothing here
+}
+
+/**
+ * Common metadata access for all elements, of simple or complex type.
+ */
+trait ElementMetadata extends TermMetadata {
+
+  /**
+   * @return the name of this element. In the case of a global/qualified name, this is only the local
+   *         part of the QName.
+   */
+  def name: String
+
+  /**
+   * @return the namespace URI as a string, or null if no namespace.
+   */
+  def namespace: String
+
+  /**
+   * @return the namespace bindings needed to construct an XML element from a Daffodil infoset
+   *         element of simple or complex type.
+   */
+  def minimizedScope: NamespaceBinding
+
+  /**
+   * @return the namespace prefix part of the XML QName of this component, or null if there
+   *         is no prefix defined or no namespace.
+   */
+  def prefix: String
+
+  /**
+   *  @return true if two or more occurrences are possible.
+   *          Note that having only 0 or 1 occurrence is not considered an array,
+   *          but rather an optional element.
+   */
+  def isArray: Boolean
+
+  /**
+   * @return true if only 0 or 1 occurrence are possible.
+   */
+  def isOptional: Boolean
+
+  /**
+   * @return the QName string for this element.
+   */
+  def toQName: String
+
+  /**
+   * @return true if the element is declared to be nillable.
+   */
+  def isNillable: Boolean
+
+  /**
+   * Provides access to the runtime properties. This is an extended collection of
+   * name-value pairs which are associated with a schema component.
+   * <p/>
+   * Runtime properties are intended to use for new ad-hoc property extensions to
+   * DFDL. These name-value pairs are visible to infoset outputters as well.
+   *
+   * @return a java-compatible map of name-value pairs.
+   */
+  def runtimeProperties: java.util.Map[String, String]
+
+}
+
+/**
+ * Access to metadata values exclusive to elements of complex type.
+ */
+trait ComplexElementMetadata extends ElementMetadata {
+
+  /**
+   * @return an ordered sequence of the child elements within this complex type regardless
+   *         of their nesting within sequence and choice groups.
+   */
+  def childMetadata: Seq[ElementMetadata]
+}
+
+/**
+ * Access to metadata values exclusive to elements of simple type.
+ */
+trait SimpleElementMetadata extends ElementMetadata {
+
+  /**
+   * The primitive type of this element as a string.
+   *
+   * @return one of boolean, double, float, date, time, dateTime, string, anyURI, hexBinary, decimal,
+   *         integer, nonNegativeInteger, byte, short, int, long, unsignedByte, unsignedShort,
+   *         unsignedInt, unsignedLong as a string. Note the initial lower-case letter in all these
+   *         type names.
+   */
+  def primTypeName: String

Review Comment:
   Why not use a enum for this? If java it would allow you to swich on it since you can't switch on a string. I think it would also give warnings if a match/case didn't cover all enum values. Maybe it's a bit annoying to convert from the internal primType objects to enum values?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import com.ibm.icu.util.Calendar
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+
+  /**
+   * @return the metadata of the element that is an array
+   */
+  override def metadata: ElementMetadata
+
+  /**
+   * @return the number of items in the array.
+   */
+  def length: Long
+
+  /**
+   * Access to an infoset element of the array.
+   * @param occursIndex1b the one-based index of the element to be accessed
+   * @return the infoset element located at the given index
+   */
+  def apply(occursIndex1b: Long): InfosetElement
+}
+
+/**
+ * API access to elements of the DFDL Infoset of both
+ * complex and simple type.
+ */
+trait InfosetElement extends InfosetItem {
+
+  /**
+   * In DFDL both simple and complex type elements can be
+   * nilled.
+   *
+   * @return true if the element is nilled, false otherwise.
+   */
+  def isNilled: Boolean
+
+  /**
+   * @return true if the element is a string or hexBinary of length 0, false otherwise.
+   */
+  def isEmpty: Boolean
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[ElementMetadata]]
+   */
+  def metadata: ElementMetadata
+
+}
+
+/**
+ * Methods specific complex elements in the infoset
+ */
+trait InfosetComplexElement extends InfosetElement {
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[ComplexElementMetadata]]
+   */
+  override def metadata: ComplexElementMetadata
+}
+
+/**
+ * Methods specific to simple elements in the infoset
+ */
+trait InfosetSimpleElement extends InfosetElement {
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[SimpleElementMetadata]]
+   */
+  override def metadata: SimpleElementMetadata
+
+  /**
+   * @return true if the element has a value, false for a nilled element.
+   */
+  def hasValue: Boolean
+
+  /**
+   * Obtains the value, then converts it to a string.
+   * Caches the string so we're not allocating strings repeatedly
+   */
+  def dataValueAsString: String
+
+  /*
+   * These are so that API users don't have to know about our
+   * very Scala-oriented DataValue type system.
+   */
+
+  /**
+   * @return the value of this simple element as a Scala AnyRef, which is
+   *         equivalent to a Java Object.
+   */
+  def getAnyRef: AnyRef
+
+  /**
+   * @return the value of this simple element as an Object (java.lang.Object),
+   *         which is equivalent to Scala AnyRef.
+   */
+  final def getObject: java.lang.Object = getAnyRef
+
+  // Note: I could not get @throws in scaladoc to work right.
+  // Complains "Could not find any member to link for ... and I tried various formulations of
+  // ClassCastException, with package, with and without [[..]].
+  // So I've just converted it to plain text.
+
+  /**
+   * @return Casts the value of this simple element as a java.math.BigDecimal.
+   * <p/>throws `ClassCastException` if the element value is not of type Decimal.
+   */
+  def getBigDecimal: JBigDecimal

Review Comment:
   A thought about all these getXYZ function is that they sortof leak internal implementation details and require infoset outputter implementations to know how daffodil internally represents the primitive types.
   
   For example, if `metadata.primTypeName` returns `unsignedInt`, then infoset outputters need to know to use `getLong` since we implement `unsignedInt` as a `Long`. It also means we can't easily change thing in the future. For example, maybe one day we decide to implement all signed integergs (byte, short, int, long) as `Long`s to simply things. That change would then leak into infoset walkers and potentially break them
   
   I'm not sure of a good way to deal with this though. Maybe the primTypeName returns the underlying type? That would mean primTypeNames would never have unsigned types, since Java has no concept of that.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetImpl.scala:
##########
@@ -1575,9 +1617,12 @@ sealed class DIComplex(override val erd: ElementRuntimeData)
   with DIComplexSharedImplMixin
   with InfosetComplexElement { diComplex =>
 
+  override def metadata: ComplexElementMetadata = erd
+
   final override def isSimple = false
   final override def isComplex = true
   final override def isArray = false
+  // final override def optArray: Option[DIArray] = None

Review Comment:
   Delete comment?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.api
+
+import scala.xml.NamespaceBinding
+
+/**
+ * This is the supportable API for access to the RuntimeData structures
+ * which provide access to static information about a given .
+ *
+ * This is used to interface other data processing fabrics to Daffodil
+ * data and metadata, by mapping to/from these metadata objects.
+ */
+trait Metadata {
+
+  /**
+   * Provides the file context of a metadata component. This refers to the specific
+   * DFDL schema file where the corresponding DFDL schema text resides corresponding
+   * to this metadata object.
+   * <p/>
+   * This is for use in diagnostic messaging. It is not the actual file URI, because
+   * those may contain personal-identifying information about the person/acccount and
+   * system that compiled the schema. It will provide enough content about the file URI that
+   * a user will be able to identify which file, but some prefix of the path
+   * components trimmed to make it of a manageable length.
+   * <p/>
+   * Used along with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]]
+   * and [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineColumnNumber]],
+   * this can give a precise location in the DFDL schema file.
+   * @return a string containing the file information, or null if unknown.
+   */
+  def schemaFileInfo: String
+
+  /**
+   * Provides the line number to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileInfo]].
+   * @return the line number as a string, or null if unknown.
+   */
+  def schemaFileLineNumber: String
+
+  /**
+   * Provides the column number within the text line, to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]].
+   * @return the column number within the text line, as a string, or null if unknown.
+   */
+  def schemaFileLineColumnNumber: String
+
+  /**
+   * The name of the schema component, in a form suitable for diagnostic messages.
+   * Unnamed components like sequence or choice groups have a diagnosticDebugName, despite not having
+   * any actual name.
+   * @return the name of the component, suitable for use in diagnostic messages.
+   */
+  def diagnosticDebugName: String
+
+  /**
+   * The standard conversion of the metadata component to a string.
+   * Unless overridden this uses the [[org.apache.daffodil.runtime1.api.Metadata.diagnosticDebugName]].
+   * @return string representation of this metadata component.
+   */
+  override def toString = diagnosticDebugName
+}
+
+/*
+ * Provides metadata access that is common to all Terms, which include
+ * Elements of simple or complex type, as well as the Sequence and Choice groups.
+ */
+trait TermMetadata extends Metadata {
+  // nothing here
+}
+
+/**
+ * Common metadata access for all elements, of simple or complex type.
+ */
+trait ElementMetadata extends TermMetadata {
+
+  /**
+   * @return the name of this element. In the case of a global/qualified name, this is only the local
+   *         part of the QName.
+   */
+  def name: String
+
+  /**
+   * @return the namespace URI as a string, or null if no namespace.
+   */
+  def namespace: String
+
+  /**
+   * @return the namespace bindings needed to construct an XML element from a Daffodil infoset
+   *         element of simple or complex type.
+   */
+  def minimizedScope: NamespaceBinding
+
+  /**
+   * @return the namespace prefix part of the XML QName of this component, or null if there
+   *         is no prefix defined or no namespace.
+   */
+  def prefix: String
+
+  /**
+   *  @return true if two or more occurrences are possible.
+   *          Note that having only 0 or 1 occurrence is not considered an array,
+   *          but rather an optional element.
+   */
+  def isArray: Boolean
+
+  /**
+   * @return true if only 0 or 1 occurrence are possible.
+   */
+  def isOptional: Boolean
+
+  /**
+   * @return the QName string for this element.
+   */
+  def toQName: String
+
+  /**
+   * @return true if the element is declared to be nillable.
+   */
+  def isNillable: Boolean
+
+  /**
+   * Provides access to the runtime properties. This is an extended collection of
+   * name-value pairs which are associated with a schema component.
+   * <p/>
+   * Runtime properties are intended to use for new ad-hoc property extensions to
+   * DFDL. These name-value pairs are visible to infoset outputters as well.
+   *
+   * @return a java-compatible map of name-value pairs.
+   */
+  def runtimeProperties: java.util.Map[String, String]
+
+}
+
+/**
+ * Access to metadata values exclusive to elements of complex type.
+ */
+trait ComplexElementMetadata extends ElementMetadata {
+
+  /**
+   * @return an ordered sequence of the child elements within this complex type regardless
+   *         of their nesting within sequence and choice groups.
+   */
+  def childMetadata: Seq[ElementMetadata]
+}
+
+/**
+ * Access to metadata values exclusive to elements of simple type.
+ */
+trait SimpleElementMetadata extends ElementMetadata {
+
+  /**
+   * The primitive type of this element as a string.
+   *
+   * @return one of boolean, double, float, date, time, dateTime, string, anyURI, hexBinary, decimal,
+   *         integer, nonNegativeInteger, byte, short, int, long, unsignedByte, unsignedShort,
+   *         unsignedInt, unsignedLong as a string. Note the initial lower-case letter in all these
+   *         type names.
+   */
+  def primTypeName: String
+}
+
+/**
+ * Access to metadata values shared by both sequences and choices
+ * which are known collectively as Model Groups.
+ */
+trait ModelGroupMetadata extends TermMetadata {}

Review Comment:
   Should ModelGroupMetaData include childMetadata like we do with complex elements?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetOutputter.scala:
##########
@@ -84,75 +85,98 @@ trait InfosetOutputter {
    *                 value, nil, name, namespace, etc.
    */
   @throws[Exception]
-  def endSimple(diSimple: DISimple): Unit
+  def endSimple(diSimple: InfosetSimpleElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of a complex element.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the complex element that is started. Various fields of
+   * @param complex the complex element that is started. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    */
   @throws[Exception]
-  def startComplex(diComplex: DIComplex): Unit
+  def startComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the end of a complex element.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the complex element that is ended. Various fields of
+   * @param complex the complex element that is ended. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    */
   @throws[Exception]
-  def endComplex(diComplex: DIComplex): Unit
+  def endComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of an array of elements.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the array that is started. Various fields of
+   * @param array the array that is started. Various fields of
    *                  DIArray can be accessed to determine things like the
    *                  name, namespace, etc.
    */
   @throws[Exception]
-  def startArray(diArray: DIArray): Unit
+  def startArray(array: InfosetArray): Unit
 
   /**
    * Called by Daffodil internals to signify the end of an array of elements.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the array that is ended. Various fields of
+   * @param array the array that is ended. Various fields of
    *                  DIArray can be accessed to determine things like the
    *                  name, namespace, etc.
    */
   @throws[Exception]
-  def endArray(diArray: DIArray): Unit
+  def endArray(array: InfosetArray): Unit
 
-  def getStatus(): Status = {
-    // Done, Ready (Not started), Visiting (part way done - can retry to visit more)...
-    status
-  }
+  def getStatus(): Status
 
   /**
-   * Helper function to determine if an element is nilled or not, taking into
-   * account whether or not the nilled state has been set yet.
+   * Override this to be fed any exceptions thrown by the start/end methods of this class.
+   * This is mostly to facilitate users debugging their code that uses the API.
    *
-   * @param diElement the element to check the nilled state of
+   * This saves the user of the API from having to wrap try/catch around their
+   * start/end method bodies.

Review Comment:
   Personally, I'm not really a huge fan of this. As annoying as wrapping function in try/catch blocks is, it's kindof the way it's done. And really, users should be handling exceptions individually instead of wrapping an entire function. That's probably bad practice.
   
   I'm also wondering if it many cases it's even too late to do anything about it and debugging wouldn't be that useful. The code and variables where the exception was thrown are out of scope, so important information might be lost. If an exception is thrown, it seems the best course of action is to look at the stack trace and point a breakpoint in the right spot. If our SDE isn't making that stacktrace available, maybe we should do that instead?
   
   That said, if we do keep this, we need to specify what a user should do with this exception in this funtion. In most cases most cases that's should probably be nothing. But I also assume they shouldn't rethrow or do anything that might cause another exception to be thrown, not sure if we handle that case.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.api
+
+import scala.xml.NamespaceBinding
+
+/**
+ * This is the supportable API for access to the RuntimeData structures
+ * which provide access to static information about a given .
+ *
+ * This is used to interface other data processing fabrics to Daffodil
+ * data and metadata, by mapping to/from these metadata objects.
+ */
+trait Metadata {
+
+  /**
+   * Provides the file context of a metadata component. This refers to the specific
+   * DFDL schema file where the corresponding DFDL schema text resides corresponding
+   * to this metadata object.
+   * <p/>
+   * This is for use in diagnostic messaging. It is not the actual file URI, because
+   * those may contain personal-identifying information about the person/acccount and
+   * system that compiled the schema. It will provide enough content about the file URI that
+   * a user will be able to identify which file, but some prefix of the path
+   * components trimmed to make it of a manageable length.
+   * <p/>
+   * Used along with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]]
+   * and [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineColumnNumber]],
+   * this can give a precise location in the DFDL schema file.
+   * @return a string containing the file information, or null if unknown.
+   */
+  def schemaFileInfo: String
+
+  /**
+   * Provides the line number to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileInfo]].
+   * @return the line number as a string, or null if unknown.
+   */
+  def schemaFileLineNumber: String
+
+  /**
+   * Provides the column number within the text line, to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]].
+   * @return the column number within the text line, as a string, or null if unknown.
+   */
+  def schemaFileLineColumnNumber: String
+
+  /**
+   * The name of the schema component, in a form suitable for diagnostic messages.
+   * Unnamed components like sequence or choice groups have a diagnosticDebugName, despite not having
+   * any actual name.
+   * @return the name of the component, suitable for use in diagnostic messages.
+   */
+  def diagnosticDebugName: String
+
+  /**
+   * The standard conversion of the metadata component to a string.
+   * Unless overridden this uses the [[org.apache.daffodil.runtime1.api.Metadata.diagnosticDebugName]].
+   * @return string representation of this metadata component.
+   */
+  override def toString = diagnosticDebugName
+}
+
+/*
+ * Provides metadata access that is common to all Terms, which include
+ * Elements of simple or complex type, as well as the Sequence and Choice groups.
+ */
+trait TermMetadata extends Metadata {
+  // nothing here
+}
+
+/**
+ * Common metadata access for all elements, of simple or complex type.
+ */
+trait ElementMetadata extends TermMetadata {
+
+  /**
+   * @return the name of this element. In the case of a global/qualified name, this is only the local
+   *         part of the QName.
+   */
+  def name: String
+
+  /**
+   * @return the namespace URI as a string, or null if no namespace.
+   */
+  def namespace: String
+
+  /**
+   * @return the namespace bindings needed to construct an XML element from a Daffodil infoset
+   *         element of simple or complex type.
+   */
+  def minimizedScope: NamespaceBinding
+
+  /**
+   * @return the namespace prefix part of the XML QName of this component, or null if there
+   *         is no prefix defined or no namespace.
+   */
+  def prefix: String
+
+  /**
+   *  @return true if two or more occurrences are possible.
+   *          Note that having only 0 or 1 occurrence is not considered an array,
+   *          but rather an optional element.
+   */
+  def isArray: Boolean
+
+  /**
+   * @return true if only 0 or 1 occurrence are possible.
+   */
+  def isOptional: Boolean
+
+  /**
+   * @return the QName string for this element.
+   */
+  def toQName: String
+
+  /**
+   * @return true if the element is declared to be nillable.
+   */
+  def isNillable: Boolean
+
+  /**
+   * Provides access to the runtime properties. This is an extended collection of
+   * name-value pairs which are associated with a schema component.
+   * <p/>
+   * Runtime properties are intended to use for new ad-hoc property extensions to
+   * DFDL. These name-value pairs are visible to infoset outputters as well.
+   *
+   * @return a java-compatible map of name-value pairs.
+   */
+  def runtimeProperties: java.util.Map[String, String]
+
+}
+
+/**
+ * Access to metadata values exclusive to elements of complex type.
+ */
+trait ComplexElementMetadata extends ElementMetadata {
+
+  /**
+   * @return an ordered sequence of the child elements within this complex type regardless
+   *         of their nesting within sequence and choice groups.
+   */
+  def childMetadata: Seq[ElementMetadata]

Review Comment:
   Why does this not include sequence and choice metadata? If we're granted access to children, it seems important to also grant access to know if those children are in hidden sequences or are in choices/unions. Personally I would prefere it we didn't grant child access at all.
   
   Can the use case of a docRoot having one complex repeating child not be figured out using the walker?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetOutputter.scala:
##########
@@ -84,75 +85,98 @@ trait InfosetOutputter {
    *                 value, nil, name, namespace, etc.
    */
   @throws[Exception]
-  def endSimple(diSimple: DISimple): Unit
+  def endSimple(diSimple: InfosetSimpleElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of a complex element.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the complex element that is started. Various fields of
+   * @param complex the complex element that is started. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    */
   @throws[Exception]
-  def startComplex(diComplex: DIComplex): Unit
+  def startComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the end of a complex element.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the complex element that is ended. Various fields of
+   * @param complex the complex element that is ended. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    */
   @throws[Exception]
-  def endComplex(diComplex: DIComplex): Unit
+  def endComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of an array of elements.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the array that is started. Various fields of
+   * @param array the array that is started. Various fields of
    *                  DIArray can be accessed to determine things like the
    *                  name, namespace, etc.
    */
   @throws[Exception]
-  def startArray(diArray: DIArray): Unit
+  def startArray(array: InfosetArray): Unit
 
   /**
    * Called by Daffodil internals to signify the end of an array of elements.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the array that is ended. Various fields of
+   * @param array the array that is ended. Various fields of
    *                  DIArray can be accessed to determine things like the
    *                  name, namespace, etc.
    */
   @throws[Exception]
-  def endArray(diArray: DIArray): Unit
+  def endArray(array: InfosetArray): Unit
 
-  def getStatus(): Status = {
-    // Done, Ready (Not started), Visiting (part way done - can retry to visit more)...
-    status
-  }
+  def getStatus(): Status
 
   /**
-   * Helper function to determine if an element is nilled or not, taking into
-   * account whether or not the nilled state has been set yet.
+   * Override this to be fed any exceptions thrown by the start/end methods of this class.
+   * This is mostly to facilitate users debugging their code that uses the API.
    *
-   * @param diElement the element to check the nilled state of
+   * This saves the user of the API from having to wrap try/catch around their
+   * start/end method bodies.
    *
-   * @return true if the nilled state has been set and is true. false if the
-   *         nilled state is false or if the nilled state has not been set yet
-   *         (e.g. during debugging)
+   * @param e the exception thrown by one of the other methods.
    */
-  final def isNilled(diElement: DIElement): Boolean = {
-    val maybeIsNilled = diElement.maybeIsNilled
-    maybeIsNilled.isDefined && maybeIsNilled.get == true
-  }
+  def exceptionOccurred(e: Exception): Unit

Review Comment:
   Sould this have a default implementation of `{}` so that infoset outputters aren't forced to implement it?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.api
+
+import scala.xml.NamespaceBinding
+
+/**
+ * This is the supportable API for access to the RuntimeData structures
+ * which provide access to static information about a given .
+ *
+ * This is used to interface other data processing fabrics to Daffodil
+ * data and metadata, by mapping to/from these metadata objects.
+ */
+trait Metadata {
+
+  /**
+   * Provides the file context of a metadata component. This refers to the specific
+   * DFDL schema file where the corresponding DFDL schema text resides corresponding
+   * to this metadata object.
+   * <p/>
+   * This is for use in diagnostic messaging. It is not the actual file URI, because
+   * those may contain personal-identifying information about the person/acccount and
+   * system that compiled the schema. It will provide enough content about the file URI that
+   * a user will be able to identify which file, but some prefix of the path
+   * components trimmed to make it of a manageable length.
+   * <p/>
+   * Used along with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]]
+   * and [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineColumnNumber]],
+   * this can give a precise location in the DFDL schema file.
+   * @return a string containing the file information, or null if unknown.
+   */
+  def schemaFileInfo: String
+
+  /**
+   * Provides the line number to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileInfo]].
+   * @return the line number as a string, or null if unknown.
+   */
+  def schemaFileLineNumber: String
+
+  /**
+   * Provides the column number within the text line, to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]].
+   * @return the column number within the text line, as a string, or null if unknown.
+   */
+  def schemaFileLineColumnNumber: String
+
+  /**
+   * The name of the schema component, in a form suitable for diagnostic messages.
+   * Unnamed components like sequence or choice groups have a diagnosticDebugName, despite not having
+   * any actual name.
+   * @return the name of the component, suitable for use in diagnostic messages.
+   */
+  def diagnosticDebugName: String

Review Comment:
   Seems odd to provide a debug thing in a public API. Should we remove this?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetOutputter.scala:
##########
@@ -190,3 +213,10 @@ object Status extends Enumeration {
   type Status = Value
   val DONE, READY, VISITING = Value
 }
+
+trait ExceptionOccurredMixin {
+  def exceptionOccurred(e: Exception): Unit = {
+    val ex = e // because sometimes the debugger won't show the exception

Review Comment:
   > // because sometimes the debugger won't show the exception
   
   Is this the Daffodil CLI debugger, or an IDE debugger? The InfosetWalker will convert any exceptions thrown by an InfosetOutputter to an SDE, maybe that is what is making it hard to see the exceptions? Maybe that should include the stack trace?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetOutputterImpl.scala:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.infoset
+
+import org.apache.daffodil.runtime1.api.InfosetElement
+
+abstract class InfosetOutputterImpl() extends BlobMethodsImpl with InfosetOutputter {
+
+  import Status._
+
+  private def status: Status = READY
+
+  override def getStatus(): Status = {
+    // Done, Ready (Not started), Visiting (part way done - can retry to visit more)...
+    status
+  }

Review Comment:
   This staus stuff doesn't even seem to be use? Can we remove it? I dont' remember what it was originally for.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/JsonInfosetOutputter.scala:
##########
@@ -92,12 +96,12 @@ class JsonInfosetOutputter private (writer: java.io.Writer, pretty: Boolean)
     if (pretty) outputIndentation(writer)
   }
 
-  override def startSimple(simple: DISimple): Unit = {
+  override def startSimple(simple: InfosetSimpleElement): Unit = {
     startNode()
     startElement(simple)
-    if (!isNilled(simple) && simple.hasValue) {
+    if (!simple.isNilled && simple.hasValue) {

Review Comment:
   I asked elsewhere if `hasValue` could be removed. Now that I look at this code, I think maybe the answer is "no", but has nothing to with nillables. I *think* it is because an infoset outputter could be use by the CLI debugger, which configures the walker to walk in places where it normally wouldn't, and so parts of an infoset might not actually have a value. So most normal infoset outputters (e.g. one uses in Drill) should never need to use `hasValue`. It's only those that could be used by the CLI debugger. I wonder if there's a better way we can handle this so that infoset outputters don't ever have to be concerned with possibly not having a value if used during debugging, and we can move that logic somewhere else?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetOutputter.scala:
##########
@@ -84,75 +85,98 @@ trait InfosetOutputter {
    *                 value, nil, name, namespace, etc.
    */
   @throws[Exception]
-  def endSimple(diSimple: DISimple): Unit
+  def endSimple(diSimple: InfosetSimpleElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of a complex element.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the complex element that is started. Various fields of
+   * @param complex the complex element that is started. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    */
   @throws[Exception]
-  def startComplex(diComplex: DIComplex): Unit
+  def startComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the end of a complex element.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the complex element that is ended. Various fields of
+   * @param complex the complex element that is ended. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    */
   @throws[Exception]
-  def endComplex(diComplex: DIComplex): Unit
+  def endComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of an array of elements.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the array that is started. Various fields of
+   * @param array the array that is started. Various fields of
    *                  DIArray can be accessed to determine things like the
    *                  name, namespace, etc.
    */
   @throws[Exception]
-  def startArray(diArray: DIArray): Unit
+  def startArray(array: InfosetArray): Unit
 
   /**
    * Called by Daffodil internals to signify the end of an array of elements.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the array that is ended. Various fields of
+   * @param array the array that is ended. Various fields of
    *                  DIArray can be accessed to determine things like the
    *                  name, namespace, etc.
    */
   @throws[Exception]
-  def endArray(diArray: DIArray): Unit
+  def endArray(array: InfosetArray): Unit
 
-  def getStatus(): Status = {
-    // Done, Ready (Not started), Visiting (part way done - can retry to visit more)...
-    status
-  }
+  def getStatus(): Status
 
   /**
-   * Helper function to determine if an element is nilled or not, taking into
-   * account whether or not the nilled state has been set yet.
+   * Override this to be fed any exceptions thrown by the start/end methods of this class.
+   * This is mostly to facilitate users debugging their code that uses the API.
    *
-   * @param diElement the element to check the nilled state of
+   * This saves the user of the API from having to wrap try/catch around their
+   * start/end method bodies.
    *
-   * @return true if the nilled state has been set and is true. false if the
-   *         nilled state is false or if the nilled state has not been set yet
-   *         (e.g. during debugging)
+   * @param e the exception thrown by one of the other methods.
    */
-  final def isNilled(diElement: DIElement): Boolean = {
-    val maybeIsNilled = diElement.maybeIsNilled
-    maybeIsNilled.isDefined && maybeIsNilled.get == true
-  }
+  def exceptionOccurred(e: Exception): Unit
+
+}
+
+/**
+ * Methods that provide Blob (Binary Large Object) support.
+ *
+ * FIXME: Scaladoc
+ */
+trait BlobMethodsMixin {
+
+  def setBlobAttributes(blobDir: Path, blobPrefix: String, blobSuffix: String): Unit
+  def setBlobPaths(empty: Seq[Path]): Unit
+  def getBlobDirectory(): Path
+  def getBlobPrefix(): String
+  def getBlobSuffix(): String
+
+}

Review Comment:
   I think I asked in the other review, but I can't find the quesiton or answer if it was answered. What is the goal of splitting BlobMethodsMixin into a separate trait instead of staying on the InofsetOutputter trait?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetOutputterImpl.scala:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.infoset
+
+import org.apache.daffodil.runtime1.api.InfosetElement
+
+abstract class InfosetOutputterImpl() extends BlobMethodsImpl with InfosetOutputter {
+
+  import Status._
+
+  private def status: Status = READY
+
+  override def getStatus(): Status = {
+    // Done, Ready (Not started), Visiting (part way done - can retry to visit more)...
+    status
+  }
+
+  /**
+   * Helper function to determine if an element is nilled or not, taking into
+   * account whether or not the nilled state has been set yet.
+   *
+   * @param elem the element to check the nilled state of
+   * @return true if the nilled state has been set and is true. false if the
+   *         nilled state is false or if the nilled state has not been set yet
+   *         (e.g. during debugging)
+   */
+  final def isNilled(elem: InfosetElement): Boolean = {
+    val diElement = elem.asInstanceOf[DIElement]
+    val maybeIsNilled = diElement.maybeIsNilled
+    maybeIsNilled.isDefined && maybeIsNilled.get == true
+  }

Review Comment:
   Can this live on the InfosetOutputter trait? It would be nice if we didn't need to have a separate InfosetOutputter and InfosetOutputterImpl classes. Seems like this is the only real differentiator.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/RuntimeData.scala:
##########
@@ -665,14 +675,29 @@ sealed class ElementRuntimeData(
     fillByteEvArg,
     maybeCheckByteAndBitOrderEvArg,
     maybeCheckBitOrderAndCharsetEvArg,
-  ) {
+  )
+  with ElementMetadata
+  with SimpleElementMetadata
+  with ComplexElementMetadata {
+
+  override def toQName: String = namedQName.toQNameString
 
   override def isRequiredScalar = !isArray && isRequiredInUnparseInfoset
 
   final def childERDs = children
 
+  final override def childMetadata: Seq[ElementRuntimeData] = childERDs

Review Comment:
   Not covered, I think this is only used by the drill walker? Any chance this could be remove and the walker could figure out what this is needed without direct access to children metadata? Maybe it's a bit more complicated, but it would be nice to minimize the API.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/unparsers/UState.scala:
##########
@@ -168,7 +168,7 @@ abstract class UState(
     Assert.invariant(Maybe.WithNulls.isDefined(currentInfosetNode))
     currentInfosetNode match {
       case a: DIArray => {
-        a.getOccurrence(arrayIterationPos)
+        a(arrayIterationPos)

Review Comment:
   Is this line not possible to hit? Is the currentInfosetNode alway a DIElement? I think it is? Can we remove this case?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/RuntimeData.scala:
##########
@@ -871,10 +901,11 @@ sealed abstract class ModelGroupRuntimeData(
     fillByteEvArg,
     maybeCheckByteAndBitOrderEvArg,
     maybeCheckBitOrderAndCharsetEvArg,
-  ) {
+  )
+  with ModelGroupMetadata {
 
   final override def isRequiredScalar = true
-  final override def isArray = false
+  final def isArray = false

Review Comment:
   Can we remove this if does override anything and isn't used?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#issuecomment-1810716932

   > +1
   > 
   > I looked at the javadoc and it doesn't include any of the new stuff since it's not in the japi project. That's expected, and doesn't have to be fixed for this PR, but we should make sure that is addressed at some point, maybe once the drill integration is complete and we are confident that the new API is sufficient.
   
   Yeah, we have to look into how to make the doc "hang together" despite some of the code being defined in java and some (most really) in scala, and some being in japi/sapi packages and some in runtime1/api package. 
   
   We may have to put in our own relative hyperlinks between the two bodies of documentation.
   
   We also really need a way to document and exclude some methods/members from the API without having to have a sharp separation between things in the sapi/japi packages which are *all* API, and things not. We need to avoid the overhead of this proxy japi/sapi structure. It's too hard to maintain.  I'd be fine with just adding to the java/scaladoc a standard comment and naming convention indicating that a method is internal not to be considered part of the API. These methods do need to be public. Even just naming them with prefix "_" and always putting them last in the trait/class/object would be ok. 
    
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on code in PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#discussion_r1387859396


##########
daffodil-japi/src/main/scala/org/apache/daffodil/japi/Daffodil.scala:
##########
@@ -514,6 +508,13 @@ class DataProcessor private[japi] (private var dp: SDataProcessor)
    */
   def save(output: WritableByteChannel): Unit = dp.save(output)
 
+  /**
+   * Walks the handler over the runtime metadata structures
+   *
+   * @param handler - the handler is called-back during the walk as each metadata structure is encountered.
+   */
+  def walkMetadata(handler: MetadataHandler) = dp.walkMetadata(handler)

Review Comment:
   Actually I thought of a better argument for why walking the runtime1 metadata is better than walking the DSOM objects.
   
   You don't need the Schema source. It works on the pre-compiled binary schema just as well as if the schema was just compiled. This bypasses the need for Daffodil's schema compiler to be involved at all in interfacing to say, Apache Drill or other data fabrics. A pre-compiled DFDL schema is all that is needed.
   
   There's a further advantage. At runtime, when you are actually parsing data, the metadata attached to the infoset objects is the runtime1 metadata objects, and the fact that those are the exact same objects you walked if you walk the metadata first in a preparation/compilation step is helpful, because at runtime it is useful to sometimes use that metadata - one would be for built-in-self-checking - make sure the metadata at run time is the expected metadata, but the other is that at runtime sometimes you actually need metadata.
   
   The example I can think of where this is useful is if a DFDL schema has a choice, and two branches of the choice both contain an element named "message", but these are of different complex types. We actually have this in a number of large message format schemas. Now, the system we are interfacing with may not have the ability for different choice branches to have children with overlapping names like this. So it may need to call these message1, message2,.... and so on. At runtime when Daffodil dispatches a startComplex event call to the InfosetOutputter, we need to look at the actual runtime metadata (ERD) of the message element to determine which one it actually is, so we can choose to populate the correct one of message1, message2, etc.
   
   This is very much like the next-element resolver that is in the runtime1 metadata. (Exposing that so it can be reused meaningfully in this case is the obvious next thought.)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on code in PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#discussion_r1391376419


##########
daffodil-core/src/test/scala/org/apache/daffodil/core/api/TestMetadataWalking.scala:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.core.api
+
+import scala.collection.mutable.ArrayBuffer
+import scala.xml.Elem
+
+import org.apache.daffodil.core.util.TestUtils
+import org.apache.daffodil.io.InputSourceDataInputStream
+import org.apache.daffodil.lib.util._
+import org.apache.daffodil.runtime1.api.ChoiceMetadata
+import org.apache.daffodil.runtime1.api.ComplexElementMetadata
+import org.apache.daffodil.runtime1.api.DFDL.ParseResult
+import org.apache.daffodil.runtime1.api.ElementMetadata
+import org.apache.daffodil.runtime1.api.InfosetArray
+import org.apache.daffodil.runtime1.api.InfosetComplexElement
+import org.apache.daffodil.runtime1.api.InfosetElement
+import org.apache.daffodil.runtime1.api.InfosetItem
+import org.apache.daffodil.runtime1.api.InfosetSimpleElement
+import org.apache.daffodil.runtime1.api.Metadata
+import org.apache.daffodil.runtime1.api.MetadataHandler
+import org.apache.daffodil.runtime1.api.SequenceMetadata
+import org.apache.daffodil.runtime1.api.SimpleElementMetadata
+import org.apache.daffodil.runtime1.infoset.ExceptionOccurredMixin
+import org.apache.daffodil.runtime1.infoset.InfosetOutputter
+import org.apache.daffodil.runtime1.infoset.InfosetOutputterImpl
+import org.apache.daffodil.runtime1.processors.DataProcessor
+
+import org.junit.Assert.assertEquals
+import org.junit.Assert.assertTrue
+import org.junit.Test
+
+class TestMetadataWalking {
+
+  def compileAndWalkMetadata(schema: Elem, mh: MetadataHandler): DataProcessor = {
+    val dp = TestUtils.compileSchema(schema)
+    assertTrue(!dp.isError)
+    dp.walkMetadata(mh)
+    dp
+  }
+
+  def parseAndWalkData(dp: DataProcessor, infosetOutputter: InfosetOutputter)(
+    data: Array[Byte],
+  ): ParseResult = {
+    val isdis = InputSourceDataInputStream(data)
+    val res = dp.parse(isdis, infosetOutputter)
+    res
+  }
+
+  class GatherMetadata extends MetadataHandler {
+
+    private val buf = new ArrayBuffer[Metadata]();
+
+    def getResult: Seq[Metadata] = {
+      val res: Seq[Metadata] = buf.toVector // makes a copy
+      buf.clear()
+      res
+    }
+
+    override def simpleElementMetadata(m: SimpleElementMetadata): Unit = buf += m
+
+    override def startComplexElementMetadata(m: ComplexElementMetadata): Unit = buf += m
+
+    override def endComplexElementMetadata(m: ComplexElementMetadata): Unit = buf += m
+
+    override def startSequenceMetadata(m: SequenceMetadata): Unit = buf += m
+
+    override def endSequenceMetadata(m: SequenceMetadata): Unit = buf += m
+
+    override def startChoiceMetadata(m: ChoiceMetadata): Unit = buf += m
+
+    override def endChoiceMetadata(m: ChoiceMetadata): Unit = buf += m

Review Comment:
   Actually, we do need the redundant name. Often I got confused between infoset event handlers and metadata event handlers when doing Drill work which uses both those APIs. That's why I added that suffix to all the names. 



##########
daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/SequenceGroup.scala:
##########
@@ -70,6 +70,11 @@ abstract class SequenceTermBase(
 
   def isOrdered: Boolean
 
+  /**
+   * Overridden in sequence group ref
+   */
+  def isHidden: Boolean = false

Review Comment:
   There are such tests already: parseSeqOfHiddenAndRegularRef  is one. 



##########
daffodil-japi/src/main/scala/org/apache/daffodil/japi/infoset/Infoset.scala:
##########
@@ -154,69 +153,69 @@ abstract class InfosetOutputter extends SInfosetOutputter {
   /**
    * Called by Daffodil internals to signify the beginning of a simple element.
    *
-   * @param diSimple the simple element that is started. Various fields of
+   * @param simple the simple element that is started. Various fields of
    *                 DISimple can be accessed to determine things like the
    *                 value, nil, name, namespace, etc.
    * @throws Exception if there was an error and Daffodil should stop parsing
    */
 
   @throws[Exception]
-  def startSimple(diSimple: DISimple): Unit
+  def startSimple(simple: InfosetSimpleElement): Unit
 
   /**
    * Called by Daffodil internals to signify the end of a simple element.
    *
-   * @param diSimple the simple element that is ended. Various fields of
+   * @param simple the simple element that is ended. Various fields of
    *                 DISimple can be accessed to determine things like the
    *                 value, nil, name, namespace, etc.
    * @throws Exception if there was an error and Daffodil should stop parsing
    */
   @throws[Exception]
-  def endSimple(diSimple: DISimple): Unit
+  def endSimple(simple: InfosetSimpleElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of a complex element.
    *
-   * @param diComplex the complex element that is started. Various fields of
+   * @param complex the complex element that is started. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    * @throws Exception if there was an error and Daffodil should stop parsing
    */
   @throws[Exception]
-  def startComplex(diComplex: DIComplex): Unit
+  def startComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the end of a complex element.
    *
-   * @param diComplex the complex element that is ended. Various fields of
+   * @param complex the complex element that is ended. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    * @throws Exception if there was an error and Daffodil should stop parsing
    */
   @throws[Exception]
-  def endComplex(diComplex: DIComplex): Unit
+  def endComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of an array of elements.
    *
-   * @param diArray the array that is started. Various fields of
+   * @param array the array that is started. Various fields of
    *                DIArray can be accessed to determine things like the
    *                name, namespace, etc.
    * @throws Exception if there was an error and Daffodil should stop parsing
    */
   @throws[Exception]
-  def startArray(diArray: DIArray): Unit
+  def startArray(array: InfosetArray): Unit
 
   /**
    * Called by Daffodil internals to signify the end of an array of elements.
    *
-   * @param diArray the array that is ended. Various fields of
+   * @param array the array that is ended. Various fields of
    *                DIArray can be accessed to determine things like the
    *                name, namespace, etc.
    * @throws Exception if there was an error and Daffodil should stop parsing
    */
   @throws[Exception]
-  def endArray(diArray: DIArray): Unit
+  def endArray(array: InfosetArray): Unit

Review Comment:
   Deprecation/Compat section added.
   We can put these added variants back in, but deprecated, at a later time if we think we need them. I don't want to clutter the API with them at this time. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.api
+
+import scala.xml.NamespaceBinding
+
+/**
+ * This is the supportable API for access to the RuntimeData structures
+ * which provide access to static information about a given .
+ *
+ * This is used to interface other data processing fabrics to Daffodil
+ * data and metadata, by mapping to/from these metadata objects.
+ */
+trait Metadata {
+
+  /**
+   * Provides the file context of a metadata component. This refers to the specific
+   * DFDL schema file where the corresponding DFDL schema text resides corresponding
+   * to this metadata object.
+   * <p/>
+   * This is for use in diagnostic messaging. It is not the actual file URI, because
+   * those may contain personal-identifying information about the person/acccount and
+   * system that compiled the schema. It will provide enough content about the file URI that
+   * a user will be able to identify which file, but some prefix of the path
+   * components trimmed to make it of a manageable length.
+   * <p/>
+   * Used along with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]]
+   * and [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineColumnNumber]],
+   * this can give a precise location in the DFDL schema file.
+   * @return a string containing the file information, or null if unknown.
+   */
+  def schemaFileInfo: String
+
+  /**
+   * Provides the line number to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileInfo]].
+   * @return the line number as a string, or null if unknown.
+   */
+  def schemaFileLineNumber: String
+
+  /**
+   * Provides the column number within the text line, to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]].
+   * @return the column number within the text line, as a string, or null if unknown.
+   */
+  def schemaFileLineColumnNumber: String
+
+  /**
+   * The name of the schema component, in a form suitable for diagnostic messages.
+   * Unnamed components like sequence or choice groups have a diagnosticDebugName, despite not having
+   * any actual name.
+   * @return the name of the component, suitable for use in diagnostic messages.
+   */
+  def diagnosticDebugName: String
+
+  /**
+   * The standard conversion of the metadata component to a string.
+   * Unless overridden this uses the [[org.apache.daffodil.runtime1.api.Metadata.diagnosticDebugName]].
+   * @return string representation of this metadata component.
+   */
+  override def toString = diagnosticDebugName
+}
+
+/*
+ * Provides metadata access that is common to all Terms, which include
+ * Elements of simple or complex type, as well as the Sequence and Choice groups.
+ */
+trait TermMetadata extends Metadata {
+  // nothing here
+}
+
+/**
+ * Common metadata access for all elements, of simple or complex type.
+ */
+trait ElementMetadata extends TermMetadata {
+
+  /**
+   * @return the name of this element. In the case of a global/qualified name, this is only the local
+   *         part of the QName.
+   */
+  def name: String
+
+  /**
+   * @return the namespace URI as a string, or null if no namespace.
+   */
+  def namespace: String
+
+  /**
+   * @return the namespace bindings needed to construct an XML element from a Daffodil infoset
+   *         element of simple or complex type.
+   */
+  def minimizedScope: NamespaceBinding
+
+  /**
+   * @return the namespace prefix part of the XML QName of this component, or null if there
+   *         is no prefix defined or no namespace.
+   */
+  def prefix: String
+
+  /**
+   *  @return true if two or more occurrences are possible.
+   *          Note that having only 0 or 1 occurrence is not considered an array,
+   *          but rather an optional element.
+   */
+  def isArray: Boolean
+
+  /**
+   * @return true if only 0 or 1 occurrence are possible.
+   */
+  def isOptional: Boolean
+
+  /**
+   * @return the QName string for this element.
+   */
+  def toQName: String
+
+  /**
+   * @return true if the element is declared to be nillable.
+   */
+  def isNillable: Boolean
+
+  /**
+   * Provides access to the runtime properties. This is an extended collection of
+   * name-value pairs which are associated with a schema component.
+   * <p/>
+   * Runtime properties are intended to use for new ad-hoc property extensions to
+   * DFDL. These name-value pairs are visible to infoset outputters as well.
+   *
+   * @return a java-compatible map of name-value pairs.
+   */
+  def runtimeProperties: java.util.Map[String, String]
+
+}
+
+/**
+ * Access to metadata values exclusive to elements of complex type.
+ */
+trait ComplexElementMetadata extends ElementMetadata {
+
+  /**
+   * @return an ordered sequence of the child elements within this complex type regardless
+   *         of their nesting within sequence and choice groups.
+   */
+  def childMetadata: Seq[ElementMetadata]
+}
+
+/**
+ * Access to metadata values exclusive to elements of simple type.
+ */
+trait SimpleElementMetadata extends ElementMetadata {
+
+  /**
+   * The primitive type of this element as a string.
+   *
+   * @return one of boolean, double, float, date, time, dateTime, string, anyURI, hexBinary, decimal,
+   *         integer, nonNegativeInteger, byte, short, int, long, unsignedByte, unsignedShort,
+   *         unsignedInt, unsignedLong as a string. Note the initial lower-case letter in all these
+   *         type names.
+   */
+  def primTypeName: String
+}
+
+/**
+ * Access to metadata values shared by both sequences and choices
+ * which are known collectively as Model Groups.
+ */
+trait ModelGroupMetadata extends TermMetadata {}

Review Comment:
   Not necessary now. Can be added in the future if needed. 
   



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.api
+
+import scala.xml.NamespaceBinding
+
+/**
+ * This is the supportable API for access to the RuntimeData structures
+ * which provide access to static information about a given .
+ *
+ * This is used to interface other data processing fabrics to Daffodil
+ * data and metadata, by mapping to/from these metadata objects.
+ */
+trait Metadata {
+
+  /**
+   * Provides the file context of a metadata component. This refers to the specific
+   * DFDL schema file where the corresponding DFDL schema text resides corresponding
+   * to this metadata object.
+   * <p/>
+   * This is for use in diagnostic messaging. It is not the actual file URI, because
+   * those may contain personal-identifying information about the person/acccount and
+   * system that compiled the schema. It will provide enough content about the file URI that
+   * a user will be able to identify which file, but some prefix of the path
+   * components trimmed to make it of a manageable length.
+   * <p/>
+   * Used along with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]]
+   * and [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineColumnNumber]],
+   * this can give a precise location in the DFDL schema file.
+   * @return a string containing the file information, or null if unknown.
+   */
+  def schemaFileInfo: String
+
+  /**
+   * Provides the line number to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileInfo]].
+   * @return the line number as a string, or null if unknown.
+   */
+  def schemaFileLineNumber: String
+
+  /**
+   * Provides the column number within the text line, to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]].
+   * @return the column number within the text line, as a string, or null if unknown.
+   */
+  def schemaFileLineColumnNumber: String
+
+  /**
+   * The name of the schema component, in a form suitable for diagnostic messages.
+   * Unnamed components like sequence or choice groups have a diagnosticDebugName, despite not having
+   * any actual name.
+   * @return the name of the component, suitable for use in diagnostic messages.
+   */
+  def diagnosticDebugName: String
+
+  /**
+   * The standard conversion of the metadata component to a string.
+   * Unless overridden this uses the [[org.apache.daffodil.runtime1.api.Metadata.diagnosticDebugName]].
+   * @return string representation of this metadata component.
+   */
+  override def toString = diagnosticDebugName
+}
+
+/*
+ * Provides metadata access that is common to all Terms, which include
+ * Elements of simple or complex type, as well as the Sequence and Choice groups.
+ */
+trait TermMetadata extends Metadata {
+  // nothing here
+}
+
+/**
+ * Common metadata access for all elements, of simple or complex type.
+ */
+trait ElementMetadata extends TermMetadata {
+
+  /**
+   * @return the name of this element. In the case of a global/qualified name, this is only the local
+   *         part of the QName.
+   */
+  def name: String
+
+  /**
+   * @return the namespace URI as a string, or null if no namespace.
+   */
+  def namespace: String
+
+  /**
+   * @return the namespace bindings needed to construct an XML element from a Daffodil infoset
+   *         element of simple or complex type.
+   */
+  def minimizedScope: NamespaceBinding
+
+  /**
+   * @return the namespace prefix part of the XML QName of this component, or null if there
+   *         is no prefix defined or no namespace.
+   */
+  def prefix: String
+
+  /**
+   *  @return true if two or more occurrences are possible.
+   *          Note that having only 0 or 1 occurrence is not considered an array,
+   *          but rather an optional element.
+   */
+  def isArray: Boolean
+
+  /**
+   * @return true if only 0 or 1 occurrence are possible.
+   */
+  def isOptional: Boolean
+
+  /**
+   * @return the QName string for this element.
+   */
+  def toQName: String
+
+  /**
+   * @return true if the element is declared to be nillable.
+   */
+  def isNillable: Boolean
+
+  /**
+   * Provides access to the runtime properties. This is an extended collection of
+   * name-value pairs which are associated with a schema component.
+   * <p/>
+   * Runtime properties are intended to use for new ad-hoc property extensions to
+   * DFDL. These name-value pairs are visible to infoset outputters as well.
+   *
+   * @return a java-compatible map of name-value pairs.
+   */
+  def runtimeProperties: java.util.Map[String, String]
+
+}
+
+/**
+ * Access to metadata values exclusive to elements of complex type.
+ */
+trait ComplexElementMetadata extends ElementMetadata {
+
+  /**
+   * @return an ordered sequence of the child elements within this complex type regardless
+   *         of their nesting within sequence and choice groups.
+   */
+  def childMetadata: Seq[ElementMetadata]

Review Comment:
   removed. 
   



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetOutputter.scala:
##########
@@ -190,3 +213,10 @@ object Status extends Enumeration {
   type Status = Value
   val DONE, READY, VISITING = Value
 }
+
+trait ExceptionOccurredMixin {
+  def exceptionOccurred(e: Exception): Unit = {
+    val ex = e // because sometimes the debugger won't show the exception

Review Comment:
   Removed



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/unparsers/UState.scala:
##########
@@ -168,7 +168,7 @@ abstract class UState(
     Assert.invariant(Maybe.WithNulls.isDefined(currentInfosetNode))
     currentInfosetNode match {
       case a: DIArray => {
-        a.getOccurrence(arrayIterationPos)
+        a(arrayIterationPos)

Review Comment:
   All related data structures take DINode, not DIElement, so I'm disinclined to change this in a major way at this time.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/RuntimeData.scala:
##########
@@ -871,10 +901,11 @@ sealed abstract class ModelGroupRuntimeData(
     fillByteEvArg,
     maybeCheckByteAndBitOrderEvArg,
     maybeCheckBitOrderAndCharsetEvArg,
-  ) {
+  )
+  with ModelGroupMetadata {
 
   final override def isRequiredScalar = true
-  final override def isArray = false
+  final def isArray = false

Review Comment:
   This is essential. Metadata walk doesn't have an array event. You get an element event for the element declaration and you call isArray for that information. 
   
   I will fix the coverage. 



##########
daffodil-japi/src/test/java/org/apache/daffodil/example/TestInfosetOutputter.java:
##########
@@ -51,45 +50,50 @@ public void endDocument() {
     }
 
     @Override
-    public void startSimple(DISimple diSimple) {
+    public void startSimple(InfosetSimpleElement diSimple) {
         events.add(
             TestInfosetEvent.startSimple(
-                diSimple.erd().name(),
-                diSimple.erd().namedQName().namespace().toString(),
+                diSimple.metadata().name(),
+                diSimple.metadata().namespace(),
                 diSimple.dataValueAsString(),
-                diSimple.erd().isNillable() ? diSimple.isNilled() : null));
+                diSimple.metadata().isNillable() ? diSimple.isNilled() : null));
     }
 
     @Override
-    public void endSimple(DISimple diSimple) {
+    public void endSimple(InfosetSimpleElement diSimple) {
         events.add(
             TestInfosetEvent.endSimple(
-                diSimple.erd().name(),
-                diSimple.erd().namedQName().namespace().toString()));
+                diSimple.metadata().name(),
+                diSimple.metadata().namespace()));
     }
 
     @Override
-    public void startComplex(DIComplex diComplex) {
+    public void startComplex(InfosetComplexElement complex) {
         events.add(
             TestInfosetEvent.startComplex(
-                diComplex.erd().name(),
-                diComplex.erd().namedQName().namespace().toString(),
-                diComplex.erd().isNillable() ? diComplex.isNilled() : null));
+                complex.metadata().name(),
+                complex.metadata().namespace(),
+                complex.metadata().isNillable() ? complex.isNilled() : null));

Review Comment:
   You do not have to call it before. isNilled will just return false if the item is non-nillable. 
   But this test is also insisting that the item is declared nillable, or at least is exercising that method for coverage reasons. 



##########
daffodil-japi/src/main/scala/org/apache/daffodil/japi/infoset/Infoset.scala:
##########
@@ -154,69 +153,69 @@ abstract class InfosetOutputter extends SInfosetOutputter {
   /**
    * Called by Daffodil internals to signify the beginning of a simple element.
    *
-   * @param diSimple the simple element that is started. Various fields of
+   * @param simple the simple element that is started. Various fields of
    *                 DISimple can be accessed to determine things like the
    *                 value, nil, name, namespace, etc.
    * @throws Exception if there was an error and Daffodil should stop parsing
    */
 
   @throws[Exception]
-  def startSimple(diSimple: DISimple): Unit
+  def startSimple(simple: InfosetSimpleElement): Unit

Review Comment:
   I would like to resolve this in a subsequent PR. 
   One thought: maybe we should provide these base traits instead as Java interfaces. They could still be used from Scala without a problem since there are no problematic data types on the calls I believe. 



##########
daffodil-core/src/test/scala/org/apache/daffodil/core/api/TestMetadataWalking.scala:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.core.api
+
+import scala.collection.mutable.ArrayBuffer
+import scala.xml.Elem
+
+import org.apache.daffodil.core.util.TestUtils
+import org.apache.daffodil.io.InputSourceDataInputStream
+import org.apache.daffodil.lib.util._
+import org.apache.daffodil.runtime1.api.ChoiceMetadata
+import org.apache.daffodil.runtime1.api.ComplexElementMetadata
+import org.apache.daffodil.runtime1.api.DFDL.ParseResult
+import org.apache.daffodil.runtime1.api.ElementMetadata
+import org.apache.daffodil.runtime1.api.InfosetArray
+import org.apache.daffodil.runtime1.api.InfosetComplexElement
+import org.apache.daffodil.runtime1.api.InfosetElement
+import org.apache.daffodil.runtime1.api.InfosetItem
+import org.apache.daffodil.runtime1.api.InfosetSimpleElement
+import org.apache.daffodil.runtime1.api.Metadata
+import org.apache.daffodil.runtime1.api.MetadataHandler
+import org.apache.daffodil.runtime1.api.SequenceMetadata
+import org.apache.daffodil.runtime1.api.SimpleElementMetadata
+import org.apache.daffodil.runtime1.infoset.ExceptionOccurredMixin
+import org.apache.daffodil.runtime1.infoset.InfosetOutputter
+import org.apache.daffodil.runtime1.infoset.InfosetOutputterImpl
+import org.apache.daffodil.runtime1.processors.DataProcessor
+
+import org.junit.Assert.assertEquals
+import org.junit.Assert.assertTrue
+import org.junit.Test
+
+class TestMetadataWalking {
+
+  def compileAndWalkMetadata(schema: Elem, mh: MetadataHandler): DataProcessor = {
+    val dp = TestUtils.compileSchema(schema)
+    assertTrue(!dp.isError)
+    dp.walkMetadata(mh)
+    dp
+  }
+
+  def parseAndWalkData(dp: DataProcessor, infosetOutputter: InfosetOutputter)(
+    data: Array[Byte],
+  ): ParseResult = {
+    val isdis = InputSourceDataInputStream(data)
+    val res = dp.parse(isdis, infosetOutputter)
+    res
+  }
+
+  class GatherMetadata extends MetadataHandler {
+
+    private val buf = new ArrayBuffer[Metadata]();
+
+    def getResult: Seq[Metadata] = {
+      val res: Seq[Metadata] = buf.toVector // makes a copy
+      buf.clear()
+      res
+    }
+
+    override def simpleElementMetadata(m: SimpleElementMetadata): Unit = buf += m
+
+    override def startComplexElementMetadata(m: ComplexElementMetadata): Unit = buf += m
+
+    override def endComplexElementMetadata(m: ComplexElementMetadata): Unit = buf += m
+
+    override def startSequenceMetadata(m: SequenceMetadata): Unit = buf += m
+
+    override def endSequenceMetadata(m: SequenceMetadata): Unit = buf += m
+
+    override def startChoiceMetadata(m: ChoiceMetadata): Unit = buf += m
+
+    override def endChoiceMetadata(m: ChoiceMetadata): Unit = buf += m
+  }
+
+  class GatherData
+    extends InfosetOutputterImpl
+    with InfosetOutputter
+    with ExceptionOccurredMixin {
+
+    private val buf = new ArrayBuffer[InfosetItem]
+
+    def getResult: Seq[InfosetItem] = {
+      val res = buf.toVector
+      reset()
+      res
+    }
+
+    override def reset(): Unit = { buf.clear() }
+
+    override def startDocument(): Unit = {}
+
+    override def endDocument(): Unit = {}

Review Comment:
   I have not needed it. I believe the concept isn't needed at all. 
   
   That is, it isn't needed in InfosetOutputter/inputter for DFDL either. Or even in the DFDL Infoset.
   
   it's a holdover from XML legacy and SAX/STAX, and the only reason it is needed there is theoretically because one could start an XML document with comments, processing-instructions, DTDs, etc. and those need a surrounding context.  For structured data where the only events are going to be associated with elements, there's no need for this extra document thing. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import com.ibm.icu.util.Calendar
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+
+  /**
+   * @return the metadata of the element that is an array
+   */
+  override def metadata: ElementMetadata
+
+  /**
+   * @return the number of items in the array.
+   */
+  def length: Long
+
+  /**
+   * Access to an infoset element of the array.
+   * @param occursIndex1b the one-based index of the element to be accessed
+   * @return the infoset element located at the given index
+   */
+  def apply(occursIndex1b: Long): InfosetElement

Review Comment:
   Unused. Removed. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import com.ibm.icu.util.Calendar
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+
+  /**
+   * @return the metadata of the element that is an array
+   */
+  override def metadata: ElementMetadata
+
+  /**
+   * @return the number of items in the array.
+   */
+  def length: Long
+
+  /**
+   * Access to an infoset element of the array.
+   * @param occursIndex1b the one-based index of the element to be accessed
+   * @return the infoset element located at the given index
+   */
+  def apply(occursIndex1b: Long): InfosetElement
+}
+
+/**
+ * API access to elements of the DFDL Infoset of both
+ * complex and simple type.
+ */
+trait InfosetElement extends InfosetItem {
+
+  /**
+   * In DFDL both simple and complex type elements can be
+   * nilled.
+   *
+   * @return true if the element is nilled, false otherwise.
+   */
+  def isNilled: Boolean
+
+  /**
+   * @return true if the element is a string or hexBinary of length 0, false otherwise.
+   */
+  def isEmpty: Boolean
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[ElementMetadata]]
+   */
+  def metadata: ElementMetadata
+
+}
+
+/**
+ * Methods specific complex elements in the infoset
+ */
+trait InfosetComplexElement extends InfosetElement {
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[ComplexElementMetadata]]
+   */
+  override def metadata: ComplexElementMetadata
+}
+
+/**
+ * Methods specific to simple elements in the infoset
+ */
+trait InfosetSimpleElement extends InfosetElement {
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[SimpleElementMetadata]]
+   */
+  override def metadata: SimpleElementMetadata
+
+  /**
+   * @return true if the element has a value, false for a nilled element.
+   */
+  def hasValue: Boolean
+
+  /**
+   * Obtains the value, then converts it to a string.
+   * Caches the string so we're not allocating strings repeatedly
+   */
+  def dataValueAsString: String

Review Comment:
   Good change. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.api
+
+import scala.xml.NamespaceBinding
+
+/**
+ * This is the supportable API for access to the RuntimeData structures
+ * which provide access to static information about a given .
+ *
+ * This is used to interface other data processing fabrics to Daffodil
+ * data and metadata, by mapping to/from these metadata objects.
+ */
+trait Metadata {
+
+  /**
+   * Provides the file context of a metadata component. This refers to the specific
+   * DFDL schema file where the corresponding DFDL schema text resides corresponding
+   * to this metadata object.
+   * <p/>
+   * This is for use in diagnostic messaging. It is not the actual file URI, because
+   * those may contain personal-identifying information about the person/acccount and
+   * system that compiled the schema. It will provide enough content about the file URI that
+   * a user will be able to identify which file, but some prefix of the path
+   * components trimmed to make it of a manageable length.
+   * <p/>
+   * Used along with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]]
+   * and [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineColumnNumber]],
+   * this can give a precise location in the DFDL schema file.
+   * @return a string containing the file information, or null if unknown.
+   */
+  def schemaFileInfo: String
+
+  /**
+   * Provides the line number to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileInfo]].
+   * @return the line number as a string, or null if unknown.
+   */
+  def schemaFileLineNumber: String
+
+  /**
+   * Provides the column number within the text line, to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]].
+   * @return the column number within the text line, as a string, or null if unknown.
+   */
+  def schemaFileLineColumnNumber: String
+
+  /**
+   * The name of the schema component, in a form suitable for diagnostic messages.
+   * Unnamed components like sequence or choice groups have a diagnosticDebugName, despite not having
+   * any actual name.
+   * @return the name of the component, suitable for use in diagnostic messages.
+   */
+  def diagnosticDebugName: String

Review Comment:
   The debug hooks have proven to be essential. This is for users of the API to debug their stuff. 
   I had to add a InfosetOutputter.exceptionOccurred(e) method so that a user can override it to access exceptions thrown by their code, as otherwise the exceptions are all being caught deep inside Daffodil, and what you get is unrelated to the cause or you have to wrap try/catch around everything in order to debug it.  



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import com.ibm.icu.util.Calendar
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+
+  /**
+   * @return the metadata of the element that is an array
+   */
+  override def metadata: ElementMetadata
+
+  /**
+   * @return the number of items in the array.
+   */
+  def length: Long
+
+  /**
+   * Access to an infoset element of the array.
+   * @param occursIndex1b the one-based index of the element to be accessed
+   * @return the infoset element located at the given index
+   */
+  def apply(occursIndex1b: Long): InfosetElement
+}
+
+/**
+ * API access to elements of the DFDL Infoset of both
+ * complex and simple type.
+ */
+trait InfosetElement extends InfosetItem {
+
+  /**
+   * In DFDL both simple and complex type elements can be
+   * nilled.
+   *
+   * @return true if the element is nilled, false otherwise.
+   */
+  def isNilled: Boolean
+
+  /**
+   * @return true if the element is a string or hexBinary of length 0, false otherwise.
+   */
+  def isEmpty: Boolean
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[ElementMetadata]]
+   */
+  def metadata: ElementMetadata
+
+}
+
+/**
+ * Methods specific complex elements in the infoset
+ */
+trait InfosetComplexElement extends InfosetElement {
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[ComplexElementMetadata]]
+   */
+  override def metadata: ComplexElementMetadata
+}
+
+/**
+ * Methods specific to simple elements in the infoset
+ */
+trait InfosetSimpleElement extends InfosetElement {
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[SimpleElementMetadata]]
+   */
+  override def metadata: SimpleElementMetadata
+
+  /**
+   * @return true if the element has a value, false for a nilled element.
+   */
+  def hasValue: Boolean
+
+  /**
+   * Obtains the value, then converts it to a string.
+   * Caches the string so we're not allocating strings repeatedly
+   */
+  def dataValueAsString: String
+
+  /*
+   * These are so that API users don't have to know about our
+   * very Scala-oriented DataValue type system.
+   */
+
+  /**
+   * @return the value of this simple element as a Scala AnyRef, which is
+   *         equivalent to a Java Object.
+   */
+  def getAnyRef: AnyRef
+
+  /**
+   * @return the value of this simple element as an Object (java.lang.Object),
+   *         which is equivalent to Scala AnyRef.
+   */
+  final def getObject: java.lang.Object = getAnyRef
+
+  // Note: I could not get @throws in scaladoc to work right.
+  // Complains "Could not find any member to link for ... and I tried various formulations of
+  // ClassCastException, with package, with and without [[..]].
+  // So I've just converted it to plain text.
+
+  /**
+   * @return Casts the value of this simple element as a java.math.BigDecimal.
+   * <p/>throws `ClassCastException` if the element value is not of type Decimal.
+   */
+  def getBigDecimal: JBigDecimal

Review Comment:
   We should change these to match the DFDL type names. The return types are not *necessarily* the exact representation types.
   ```
   def getUnsignedInt: Long 
   ```
   Requires only that you know the primTypeName to know what to call, and the return type can be kept the same regardless of the implementation.
   
   People always expect getInt will work for smaller integer types also, so I've changed it to also convert from
   
   getAnyRef and getObject should return the same types that the other getTYPE functions do. So for example if the data is a date type, getDate returns a Calendar, getAnyRef should return a Calendar also for that same element. 
   
   I will make this change. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import com.ibm.icu.util.Calendar
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+
+  /**
+   * @return the metadata of the element that is an array
+   */
+  override def metadata: ElementMetadata
+
+  /**
+   * @return the number of items in the array.
+   */
+  def length: Long
+
+  /**
+   * Access to an infoset element of the array.
+   * @param occursIndex1b the one-based index of the element to be accessed
+   * @return the infoset element located at the given index
+   */
+  def apply(occursIndex1b: Long): InfosetElement
+}
+
+/**
+ * API access to elements of the DFDL Infoset of both
+ * complex and simple type.
+ */
+trait InfosetElement extends InfosetItem {
+
+  /**
+   * In DFDL both simple and complex type elements can be
+   * nilled.
+   *
+   * @return true if the element is nilled, false otherwise.
+   */
+  def isNilled: Boolean
+
+  /**
+   * @return true if the element is a string or hexBinary of length 0, false otherwise.
+   */
+  def isEmpty: Boolean
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[ElementMetadata]]
+   */
+  def metadata: ElementMetadata
+
+}
+
+/**
+ * Methods specific complex elements in the infoset
+ */
+trait InfosetComplexElement extends InfosetElement {
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[ComplexElementMetadata]]
+   */
+  override def metadata: ComplexElementMetadata
+}
+
+/**
+ * Methods specific to simple elements in the infoset
+ */
+trait InfosetSimpleElement extends InfosetElement {
+
+  /*
+   * Access to the metadata information about this element.
+   * See [[SimpleElementMetadata]]
+   */
+  override def metadata: SimpleElementMetadata
+
+  /**
+   * @return true if the element has a value, false for a nilled element.
+   */
+  def hasValue: Boolean

Review Comment:
   Within daffodil internals these are different because there are states like where we don't know if the element will be setNilled or not and we want to know if it has a value. 
   
   But at this API, the distinction is not needed, as simple elements are finalized before they're walked. 
   
   So this can be removed. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetOutputter.scala:
##########
@@ -84,75 +85,98 @@ trait InfosetOutputter {
    *                 value, nil, name, namespace, etc.
    */
   @throws[Exception]
-  def endSimple(diSimple: DISimple): Unit
+  def endSimple(diSimple: InfosetSimpleElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of a complex element.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the complex element that is started. Various fields of
+   * @param complex the complex element that is started. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    */
   @throws[Exception]
-  def startComplex(diComplex: DIComplex): Unit
+  def startComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the end of a complex element.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the complex element that is ended. Various fields of
+   * @param complex the complex element that is ended. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    */
   @throws[Exception]
-  def endComplex(diComplex: DIComplex): Unit
+  def endComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of an array of elements.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the array that is started. Various fields of
+   * @param array the array that is started. Various fields of
    *                  DIArray can be accessed to determine things like the
    *                  name, namespace, etc.
    */
   @throws[Exception]
-  def startArray(diArray: DIArray): Unit
+  def startArray(array: InfosetArray): Unit
 
   /**
    * Called by Daffodil internals to signify the end of an array of elements.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the array that is ended. Various fields of
+   * @param array the array that is ended. Various fields of
    *                  DIArray can be accessed to determine things like the
    *                  name, namespace, etc.
    */
   @throws[Exception]
-  def endArray(diArray: DIArray): Unit
+  def endArray(array: InfosetArray): Unit
 
-  def getStatus(): Status = {
-    // Done, Ready (Not started), Visiting (part way done - can retry to visit more)...
-    status
-  }
+  def getStatus(): Status
 
   /**
-   * Helper function to determine if an element is nilled or not, taking into
-   * account whether or not the nilled state has been set yet.
+   * Override this to be fed any exceptions thrown by the start/end methods of this class.
+   * This is mostly to facilitate users debugging their code that uses the API.
    *
-   * @param diElement the element to check the nilled state of
+   * This saves the user of the API from having to wrap try/catch around their
+   * start/end method bodies.

Review Comment:
   Removed. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import com.ibm.icu.util.Calendar
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+
+  /**
+   * @return the metadata of the element that is an array
+   */
+  override def metadata: ElementMetadata
+
+  /**
+   * @return the number of items in the array.
+   */
+  def length: Long
+
+  /**
+   * Access to an infoset element of the array.
+   * @param occursIndex1b the one-based index of the element to be accessed
+   * @return the infoset element located at the given index
+   */
+  def apply(occursIndex1b: Long): InfosetElement
+}
+
+/**
+ * API access to elements of the DFDL Infoset of both
+ * complex and simple type.
+ */
+trait InfosetElement extends InfosetItem {
+
+  /**
+   * In DFDL both simple and complex type elements can be
+   * nilled.
+   *
+   * @return true if the element is nilled, false otherwise.
+   */
+  def isNilled: Boolean
+
+  /**
+   * @return true if the element is a string or hexBinary of length 0, false otherwise.

Review Comment:
   unused. removed. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetOutputter.scala:
##########
@@ -84,75 +85,98 @@ trait InfosetOutputter {
    *                 value, nil, name, namespace, etc.
    */
   @throws[Exception]
-  def endSimple(diSimple: DISimple): Unit
+  def endSimple(diSimple: InfosetSimpleElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of a complex element.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the complex element that is started. Various fields of
+   * @param complex the complex element that is started. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    */
   @throws[Exception]
-  def startComplex(diComplex: DIComplex): Unit
+  def startComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the end of a complex element.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the complex element that is ended. Various fields of
+   * @param complex the complex element that is ended. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    */
   @throws[Exception]
-  def endComplex(diComplex: DIComplex): Unit
+  def endComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of an array of elements.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the array that is started. Various fields of
+   * @param array the array that is started. Various fields of
    *                  DIArray can be accessed to determine things like the
    *                  name, namespace, etc.
    */
   @throws[Exception]
-  def startArray(diArray: DIArray): Unit
+  def startArray(array: InfosetArray): Unit
 
   /**
    * Called by Daffodil internals to signify the end of an array of elements.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the array that is ended. Various fields of
+   * @param array the array that is ended. Various fields of
    *                  DIArray can be accessed to determine things like the
    *                  name, namespace, etc.
    */
   @throws[Exception]
-  def endArray(diArray: DIArray): Unit
+  def endArray(array: InfosetArray): Unit
 
-  def getStatus(): Status = {
-    // Done, Ready (Not started), Visiting (part way done - can retry to visit more)...
-    status
-  }
+  def getStatus(): Status
 
   /**
-   * Helper function to determine if an element is nilled or not, taking into
-   * account whether or not the nilled state has been set yet.
+   * Override this to be fed any exceptions thrown by the start/end methods of this class.
+   * This is mostly to facilitate users debugging their code that uses the API.
    *
-   * @param diElement the element to check the nilled state of
+   * This saves the user of the API from having to wrap try/catch around their
+   * start/end method bodies.
    *
-   * @return true if the nilled state has been set and is true. false if the
-   *         nilled state is false or if the nilled state has not been set yet
-   *         (e.g. during debugging)
+   * @param e the exception thrown by one of the other methods.
    */
-  final def isNilled(diElement: DIElement): Boolean = {
-    val maybeIsNilled = diElement.maybeIsNilled
-    maybeIsNilled.isDefined && maybeIsNilled.get == true
-  }
+  def exceptionOccurred(e: Exception): Unit
+
+}
+
+/**
+ * Methods that provide Blob (Binary Large Object) support.
+ *
+ * FIXME: Scaladoc
+ */
+trait BlobMethodsMixin {
+
+  def setBlobAttributes(blobDir: Path, blobPrefix: String, blobSuffix: String): Unit
+  def setBlobPaths(empty: Seq[Path]): Unit
+  def getBlobDirectory(): Path
+  def getBlobPrefix(): String
+  def getBlobSuffix(): String
+
+}

Review Comment:
   Since it has one constant implementation for all InfosetOutputters, it isn't really part of the API.  Users neither need to override any of it, nor call any of it. 
   
   That said, I can elimiinate the trait,and since InfosetOutputterImpl is going away it just becomes a trait mixed into InfosetOutputter. 
   
   



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetOutputter.scala:
##########
@@ -84,75 +85,98 @@ trait InfosetOutputter {
    *                 value, nil, name, namespace, etc.
    */
   @throws[Exception]
-  def endSimple(diSimple: DISimple): Unit
+  def endSimple(diSimple: InfosetSimpleElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of a complex element.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the complex element that is started. Various fields of
+   * @param complex the complex element that is started. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    */
   @throws[Exception]
-  def startComplex(diComplex: DIComplex): Unit
+  def startComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the end of a complex element.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the complex element that is ended. Various fields of
+   * @param complex the complex element that is ended. Various fields of
    *                  DIComplex can be accessed to determine things like the
    *                  nil, name, namespace, etc.
    */
   @throws[Exception]
-  def endComplex(diComplex: DIComplex): Unit
+  def endComplex(complex: InfosetComplexElement): Unit
 
   /**
    * Called by Daffodil internals to signify the beginning of an array of elements.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the array that is started. Various fields of
+   * @param array the array that is started. Various fields of
    *                  DIArray can be accessed to determine things like the
    *                  name, namespace, etc.
    */
   @throws[Exception]
-  def startArray(diArray: DIArray): Unit
+  def startArray(array: InfosetArray): Unit
 
   /**
    * Called by Daffodil internals to signify the end of an array of elements.
    *
    * Throws java.lang.Exception if there was an error and Daffodil should stop parsing
    *
-   * @param diComplex the array that is ended. Various fields of
+   * @param array the array that is ended. Various fields of
    *                  DIArray can be accessed to determine things like the
    *                  name, namespace, etc.
    */
   @throws[Exception]
-  def endArray(diArray: DIArray): Unit
+  def endArray(array: InfosetArray): Unit
 
-  def getStatus(): Status = {
-    // Done, Ready (Not started), Visiting (part way done - can retry to visit more)...
-    status
-  }
+  def getStatus(): Status
 
   /**
-   * Helper function to determine if an element is nilled or not, taking into
-   * account whether or not the nilled state has been set yet.
+   * Override this to be fed any exceptions thrown by the start/end methods of this class.
+   * This is mostly to facilitate users debugging their code that uses the API.
    *
-   * @param diElement the element to check the nilled state of
+   * This saves the user of the API from having to wrap try/catch around their
+   * start/end method bodies.
    *
-   * @return true if the nilled state has been set and is true. false if the
-   *         nilled state is false or if the nilled state has not been set yet
-   *         (e.g. during debugging)
+   * @param e the exception thrown by one of the other methods.
    */
-  final def isNilled(diElement: DIElement): Boolean = {
-    val maybeIsNilled = diElement.maybeIsNilled
-    maybeIsNilled.isDefined && maybeIsNilled.get == true
-  }
+  def exceptionOccurred(e: Exception): Unit

Review Comment:
   Removed.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetOutputterImpl.scala:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.infoset
+
+import org.apache.daffodil.runtime1.api.InfosetElement
+
+abstract class InfosetOutputterImpl() extends BlobMethodsImpl with InfosetOutputter {
+
+  import Status._
+
+  private def status: Status = READY
+
+  override def getStatus(): Status = {
+    // Done, Ready (Not started), Visiting (part way done - can retry to visit more)...
+    status
+  }
+
+  /**
+   * Helper function to determine if an element is nilled or not, taking into
+   * account whether or not the nilled state has been set yet.
+   *
+   * @param elem the element to check the nilled state of
+   * @return true if the nilled state has been set and is true. false if the
+   *         nilled state is false or if the nilled state has not been set yet
+   *         (e.g. during debugging)
+   */
+  final def isNilled(elem: InfosetElement): Boolean = {
+    val diElement = elem.asInstanceOf[DIElement]
+    val maybeIsNilled = diElement.maybeIsNilled
+    maybeIsNilled.isDefined && maybeIsNilled.get == true
+  }

Review Comment:
   removed isNilled method (only needed in one place)



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/JsonInfosetOutputter.scala:
##########
@@ -112,11 +116,12 @@ class JsonInfosetOutputter private (writer: java.io.Writer, pretty: Boolean)
     }
   }
 
-  override def endSimple(simple: DISimple): Unit = {
+  override def endSimple(se: InfosetSimpleElement): Unit = {
     // nothing to do
   }
 
-  override def startComplex(complex: DIComplex): Unit = {
+  override def startComplex(ce: InfosetComplexElement): Unit = {
+    val complex = ce.asInstanceOf[DIComplex]

Review Comment:
   fixed



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/JsonInfosetOutputter.scala:
##########
@@ -92,12 +96,12 @@ class JsonInfosetOutputter private (writer: java.io.Writer, pretty: Boolean)
     if (pretty) outputIndentation(writer)
   }
 
-  override def startSimple(simple: DISimple): Unit = {
+  override def startSimple(simple: InfosetSimpleElement): Unit = {
     startNode()
     startElement(simple)
-    if (!isNilled(simple) && simple.hasValue) {
+    if (!simple.isNilled && simple.hasValue) {

Review Comment:
   I removed hasValue from the api.InfosetSimpleElement trait. 
   
   We do need it, but it can be on DISimple only 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/RuntimeData.scala:
##########
@@ -665,14 +675,29 @@ sealed class ElementRuntimeData(
     fillByteEvArg,
     maybeCheckByteAndBitOrderEvArg,
     maybeCheckBitOrderAndCharsetEvArg,
-  ) {
+  )
+  with ElementMetadata
+  with SimpleElementMetadata
+  with ComplexElementMetadata {
+
+  override def toQName: String = namedQName.toQNameString
 
   override def isRequiredScalar = !isArray && isRequiredInUnparseInfoset
 
   final def childERDs = children
 
+  final override def childMetadata: Seq[ElementRuntimeData] = childERDs

Review Comment:
   removing 
    I don't want to have to filter it on isHidden of enclosing sequences. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on code in PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#discussion_r1390326486


##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,274 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.api
+
+import scala.xml.NamespaceBinding
+
+import org.apache.daffodil.lib.xml.NamedQName
+import org.apache.daffodil.runtime1.dpath.PrimTypeNode
+
+/**
+ * This is the supportable API for access to the RuntimeData structures
+ * which provide access to static information about a given .
+ *
+ * This is used to interface other data processing fabrics to Daffodil
+ * data and metadata, by mapping to/from these metadata objects.
+ */
+trait Metadata {
+
+  /**
+   * Provides the file context of a metadata component. This refers to the specific
+   * DFDL schema file where the corresponding DFDL schema text resides corresponding
+   * to this metadata object.
+   * <p/>
+   * This is for use in diagnostic messaging. It is not the actual file URI, because
+   * those may contain personal-identifying information about the person/acccount and
+   * system that compiled the schema. It will provide enough content about the file URI that
+   * a user will be able to identify which file, but some prefix of the path
+   * components trimmed to make it of a manageable length.
+   * <p/>
+   * Used along with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]]
+   * and [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineColumnNumber]],
+   * this can give a precise location in the DFDL schema file.
+   * @return a string containing the file information, or null if unknown.
+   */
+  def schemaFileInfo: String
+
+  /**
+   * Provides the line number to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileInfo]].
+   * @return the line number as a string, or null if unknown.
+   */
+  def schemaFileLineNumber: String
+
+  /**
+   * Provides the column number within the text line, to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]].
+   * @return the column number within the text line, as a string, or null if unknown.
+   */
+  def schemaFileLineColumnNumber: String
+
+  def path: String
+
+  /**
+   * The name of the schema component, in a form suitable for diagnostic messages.
+   * Unnamed components like sequence or choice groups have a diagnosticDebugName, despite not having
+   * any actual name.
+   * @return the name of the component, suitable for use in diagnostic messages.
+   */
+  def diagnosticDebugName: String
+
+  /**
+   * The standard conversion of the metadata component to a string.
+   * Unless overridden this uses the [[org.apache.daffodil.runtime1.api.Metadata.diagnosticDebugName]].
+   * @return string representation of this metadata component.
+   */
+  override def toString = diagnosticDebugName
+}
+
+/*
+ * Provides metadata access that is common to all Terms, which include
+ * Elements of simple or complex type, as well as the Sequence and Choice groups.
+ */
+trait TermMetadata extends Metadata {
+  // nothing here
+}
+
+/**
+ * Common metadata access for all elements, of simple or complex type.
+ */
+trait ElementMetadata extends TermMetadata {
+
+  /**
+   * @return the name of this element. In the case of a global/qualified name, this is only the local
+   *         part of the QName.
+   */
+  def name: String
+
+  /**
+   * @return the namespace URI as a string, or null if no namespace.
+   */
+  def namespace: String
+
+  /**
+   * @return the namespace bindings needed to construct an XML element from a Daffodil infoset
+   *         element of simple or complex type.
+   */
+  def minimizedScope: NamespaceBinding
+
+  /**
+   * @return the namespace prefix part of the XML QName of this component, or null if there
+   *         is no prefix defined or no namespace.
+   */
+  def prefix: String
+
+  /**
+   *  @return true if two or more occurrences are possible.
+   *          Note that having only 0 or 1 occurrence is not considered an array,
+   *          but rather an optional element.
+   */
+  def isArray: Boolean
+
+  /**
+   * @return true if only 0 or 1 occurrence are possible.
+   */
+  def isOptional: Boolean
+
+  /**
+   * @return
+   */
+  def namedQName: NamedQName

Review Comment:
   This should go away.  The only added method needed is toQName which returns a string and we can get this class out of the API. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,274 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.api
+
+import scala.xml.NamespaceBinding
+
+import org.apache.daffodil.lib.xml.NamedQName
+import org.apache.daffodil.runtime1.dpath.PrimTypeNode
+
+/**
+ * This is the supportable API for access to the RuntimeData structures
+ * which provide access to static information about a given .
+ *
+ * This is used to interface other data processing fabrics to Daffodil
+ * data and metadata, by mapping to/from these metadata objects.
+ */
+trait Metadata {
+
+  /**
+   * Provides the file context of a metadata component. This refers to the specific
+   * DFDL schema file where the corresponding DFDL schema text resides corresponding
+   * to this metadata object.
+   * <p/>
+   * This is for use in diagnostic messaging. It is not the actual file URI, because
+   * those may contain personal-identifying information about the person/acccount and
+   * system that compiled the schema. It will provide enough content about the file URI that
+   * a user will be able to identify which file, but some prefix of the path
+   * components trimmed to make it of a manageable length.
+   * <p/>
+   * Used along with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]]
+   * and [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineColumnNumber]],
+   * this can give a precise location in the DFDL schema file.
+   * @return a string containing the file information, or null if unknown.
+   */
+  def schemaFileInfo: String
+
+  /**
+   * Provides the line number to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileInfo]].
+   * @return the line number as a string, or null if unknown.
+   */
+  def schemaFileLineNumber: String
+
+  /**
+   * Provides the column number within the text line, to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]].
+   * @return the column number within the text line, as a string, or null if unknown.
+   */
+  def schemaFileLineColumnNumber: String
+
+  def path: String
+
+  /**
+   * The name of the schema component, in a form suitable for diagnostic messages.
+   * Unnamed components like sequence or choice groups have a diagnosticDebugName, despite not having
+   * any actual name.
+   * @return the name of the component, suitable for use in diagnostic messages.
+   */
+  def diagnosticDebugName: String
+
+  /**
+   * The standard conversion of the metadata component to a string.
+   * Unless overridden this uses the [[org.apache.daffodil.runtime1.api.Metadata.diagnosticDebugName]].
+   * @return string representation of this metadata component.
+   */
+  override def toString = diagnosticDebugName
+}
+
+/*
+ * Provides metadata access that is common to all Terms, which include
+ * Elements of simple or complex type, as well as the Sequence and Choice groups.
+ */
+trait TermMetadata extends Metadata {
+  // nothing here
+}
+
+/**
+ * Common metadata access for all elements, of simple or complex type.
+ */
+trait ElementMetadata extends TermMetadata {
+
+  /**
+   * @return the name of this element. In the case of a global/qualified name, this is only the local
+   *         part of the QName.
+   */
+  def name: String
+
+  /**
+   * @return the namespace URI as a string, or null if no namespace.
+   */
+  def namespace: String
+
+  /**
+   * @return the namespace bindings needed to construct an XML element from a Daffodil infoset
+   *         element of simple or complex type.
+   */
+  def minimizedScope: NamespaceBinding
+
+  /**
+   * @return the namespace prefix part of the XML QName of this component, or null if there
+   *         is no prefix defined or no namespace.
+   */
+  def prefix: String
+
+  /**
+   *  @return true if two or more occurrences are possible.
+   *          Note that having only 0 or 1 occurrence is not considered an array,
+   *          but rather an optional element.
+   */
+  def isArray: Boolean
+
+  /**
+   * @return true if only 0 or 1 occurrence are possible.
+   */
+  def isOptional: Boolean
+
+  /**
+   * @return
+   */
+  def namedQName: NamedQName
+
+  /**
+   * @return true if the element is declared to be nillable.
+   */
+  def isNillable: Boolean
+
+  /**
+   * Provides access to the runtime properties. This is an extended collection of
+   * name-value pairs which are associated with a schema component.
+   * <p/>
+   * Runtime properties are intended to use for new ad-hoc property extensions to
+   * DFDL. These name-value pairs are visible to infoset outputters as well.
+   *
+   * @return a java-compatible map of name-value pairs.
+   */
+  def runtimeProperties: java.util.Map[String, String]
+
+}
+
+/**
+ * Access to metadata values exclusive to elements of complex type.
+ */
+trait ComplexElementMetadata extends ElementMetadata {
+
+  /**
+   * @return an ordered sequence of the child elements within this complex type regardless
+   *         of their nesting within sequence and choice groups.
+   */
+  def childMetadata: Seq[ElementMetadata]
+}
+
+/**
+ * Access to metadata values exclusive to elements of simple type.
+ */
+trait SimpleElementMetadata extends ElementMetadata {
+
+  /**
+   * The primitive type of this element as a string.
+   *
+   * @return one of Boolean, Double, Float, Date, Time, DateTime, String, AnyURI, HexBinary, Decimal,
+   *         Integer, NonNegativeInteger, Byte, Short, Int, Long, UnsignedByte, UnsignedShort,
+   *         UnsignedInt, UnsignedLong as a string. Note the initial upper-case letter in all these
+   *         type names.
+   */
+  def primType: PrimTypeNode

Review Comment:
   This method is also problematic. It exposes PrimTypeNode, which we don't want on our API. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,274 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.api
+
+import scala.xml.NamespaceBinding
+
+import org.apache.daffodil.lib.xml.NamedQName
+import org.apache.daffodil.runtime1.dpath.PrimTypeNode
+
+/**
+ * This is the supportable API for access to the RuntimeData structures
+ * which provide access to static information about a given .
+ *
+ * This is used to interface other data processing fabrics to Daffodil
+ * data and metadata, by mapping to/from these metadata objects.
+ */
+trait Metadata {
+
+  /**
+   * Provides the file context of a metadata component. This refers to the specific
+   * DFDL schema file where the corresponding DFDL schema text resides corresponding
+   * to this metadata object.
+   * <p/>
+   * This is for use in diagnostic messaging. It is not the actual file URI, because
+   * those may contain personal-identifying information about the person/acccount and
+   * system that compiled the schema. It will provide enough content about the file URI that
+   * a user will be able to identify which file, but some prefix of the path
+   * components trimmed to make it of a manageable length.
+   * <p/>
+   * Used along with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]]
+   * and [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineColumnNumber]],
+   * this can give a precise location in the DFDL schema file.
+   * @return a string containing the file information, or null if unknown.
+   */
+  def schemaFileInfo: String
+
+  /**
+   * Provides the line number to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileInfo]].
+   * @return the line number as a string, or null if unknown.
+   */
+  def schemaFileLineNumber: String
+
+  /**
+   * Provides the column number within the text line, to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]].
+   * @return the column number within the text line, as a string, or null if unknown.
+   */
+  def schemaFileLineColumnNumber: String
+
+  def path: String

Review Comment:
   Needs scaladoc



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,274 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.api
+
+import scala.xml.NamespaceBinding
+
+import org.apache.daffodil.lib.xml.NamedQName
+import org.apache.daffodil.runtime1.dpath.PrimTypeNode
+
+/**
+ * This is the supportable API for access to the RuntimeData structures
+ * which provide access to static information about a given .
+ *
+ * This is used to interface other data processing fabrics to Daffodil
+ * data and metadata, by mapping to/from these metadata objects.
+ */
+trait Metadata {
+
+  /**
+   * Provides the file context of a metadata component. This refers to the specific
+   * DFDL schema file where the corresponding DFDL schema text resides corresponding
+   * to this metadata object.
+   * <p/>
+   * This is for use in diagnostic messaging. It is not the actual file URI, because
+   * those may contain personal-identifying information about the person/acccount and
+   * system that compiled the schema. It will provide enough content about the file URI that
+   * a user will be able to identify which file, but some prefix of the path
+   * components trimmed to make it of a manageable length.
+   * <p/>
+   * Used along with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]]
+   * and [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineColumnNumber]],
+   * this can give a precise location in the DFDL schema file.
+   * @return a string containing the file information, or null if unknown.
+   */
+  def schemaFileInfo: String
+
+  /**
+   * Provides the line number to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileInfo]].
+   * @return the line number as a string, or null if unknown.
+   */
+  def schemaFileLineNumber: String
+
+  /**
+   * Provides the column number within the text line, to go with [[org.apache.daffodil.runtime1.api.Metadata.schemaFileLineNumber]].
+   * @return the column number within the text line, as a string, or null if unknown.
+   */
+  def schemaFileLineColumnNumber: String
+
+  def path: String
+
+  /**
+   * The name of the schema component, in a form suitable for diagnostic messages.
+   * Unnamed components like sequence or choice groups have a diagnosticDebugName, despite not having
+   * any actual name.
+   * @return the name of the component, suitable for use in diagnostic messages.
+   */
+  def diagnosticDebugName: String
+
+  /**
+   * The standard conversion of the metadata component to a string.
+   * Unless overridden this uses the [[org.apache.daffodil.runtime1.api.Metadata.diagnosticDebugName]].
+   * @return string representation of this metadata component.
+   */
+  override def toString = diagnosticDebugName
+}
+
+/*
+ * Provides metadata access that is common to all Terms, which include
+ * Elements of simple or complex type, as well as the Sequence and Choice groups.
+ */
+trait TermMetadata extends Metadata {
+  // nothing here
+}
+
+/**
+ * Common metadata access for all elements, of simple or complex type.
+ */
+trait ElementMetadata extends TermMetadata {
+
+  /**
+   * @return the name of this element. In the case of a global/qualified name, this is only the local
+   *         part of the QName.
+   */
+  def name: String
+
+  /**
+   * @return the namespace URI as a string, or null if no namespace.
+   */
+  def namespace: String
+
+  /**
+   * @return the namespace bindings needed to construct an XML element from a Daffodil infoset
+   *         element of simple or complex type.
+   */
+  def minimizedScope: NamespaceBinding

Review Comment:
   Everything one needs to synthesize XML as an output representation is needed, and this was one of the things. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/MetadataWalker.scala:
##########
@@ -0,0 +1,76 @@
+package org.apache.daffodil.runtime1.processors

Review Comment:
   Ok, so this metadata walker is centralized in runtime1. SAPI/JAPI just have a walkMetadata method that takes a MetadataHandler. The runtime1/api/MetadataHandler class is perhaps the one to be concerned about. It has two methods that I don't like. Comments on that file. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#issuecomment-1809939347

   @stevedlawrence @tuxji  This should be very close to ready now. 
   Please give this a review again. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#issuecomment-1807006933

   @stevedlawrence This is ready for a next review pass. I've addressed all the prior comments I believe, and cleaned up several other things. 
   
   Once merged, I want to run a large schema regression test with as many schemas as possible to verify this hasn't broken things. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on code in PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#discussion_r1392925802


##########
daffodil-japi/src/test/java/org/apache/daffodil/example/TestJavaAPI.java:
##########
@@ -1209,7 +1209,6 @@ public void testJavaAPI25() throws IOException, ClassNotFoundException, External
         assertEquals(expectedData, bos.toString());
     }
 
-    @Test

Review Comment:
   Looks like a mistake. I'll add this back. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Long => JLong }
+import scala.collection.JavaConverters._
+import scala.xml.NamespaceBinding
+
+import org.apache.daffodil.runtime1.dpath.NodeInfo
+import org.apache.daffodil.runtime1.dpath.NodeInfo.PrimType
+
+/**
+ * This is the supportable API for access to the RuntimeData structures
+ * which provide access to static information about a given .

Review Comment:
   will fix. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["stevedlawrence (via GitHub)" <gi...@apache.org>]

stevedlawrence commented on code in PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#discussion_r1387930288


##########
daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/SequenceTermRuntime1Mixin.scala:
##########
@@ -49,6 +50,10 @@ trait SequenceTermRuntime1Mixin { self: SequenceTermBase =>
       fillByteEv,
       maybeCheckByteAndBitOrderEv,
       maybeCheckBitOrderAndCharsetEv,
+      this match {
+        case sgr: SequenceGroupRef => sgr.isHidden
+        case _ => false
+      },

Review Comment:
   Is it worth adding an `isHidden` member to `SequenceTermBase` that defaults to false, and `SequenceGroupRef` overrides it? So this match case just becomes `isHidden`? Maybe that refactor is best done in another PR.



##########
daffodil-japi/src/main/scala/org/apache/daffodil/japi/Daffodil.scala:
##########
@@ -514,6 +508,13 @@ class DataProcessor private[japi] (private var dp: SDataProcessor)
    */
   def save(output: WritableByteChannel): Unit = dp.save(output)
 
+  /**
+   * Walks the handler over the runtime metadata structures
+   *
+   * @param handler - the handler is called-back during the walk as each metadata structure is encountered.
+   */
+  def walkMetadata(handler: MetadataHandler) = dp.walkMetadata(handler)

Review Comment:
   Can you add `: Unit` to this to make it obvious this doesn't return anything?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement

Review Comment:
   Do API users need access to the `parent`?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)

Review Comment:
   Have you looked at what is generated for JavaDoc to make sure it seems reasonable? I imagine this will all end up in the JAPI, it would be good to verify it looks sane.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement
+  def isNilled: Boolean
+  def isEmpty: Boolean
+  def asSimple: InfosetSimpleElement
+  def asComplex: InfosetComplexElement

Review Comment:
   What happens if a API user calls `asSimple` when the element is a complex, or vice versa? Do users even ever get a `InfosetElement`? Seems they should always get a `InfosetSimpleElement` or an `InfosetComplexElement` so these methods should be needed?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetImpl.scala:
##########
@@ -1558,6 +1590,29 @@ sealed class DISimple(override val erd: ElementRuntimeData)
     Assert.invariantFailed("Should not try to remove a child of a simple type")
   }
 
+  // call to getAnyRef here seems redundant
+  // but tests fail that compare objects if this call isn't here
+  // Boxing/Unboxing related perhaps
+  @inline private def v = dataValue.getAnyRef
+
+  override def getAnyRef: AnyRef = v.asInstanceOf[AnyRef]
+  override def getBigDecimal: JBigDecimal = v.asInstanceOf[JBigDecimal]
+  override def getCalendar: DFDLCalendar = v.asInstanceOf[DFDLCalendar]
+  override def getDate: DFDLDate = v.asInstanceOf[DFDLDate]
+  override def getTime: DFDLTime = v.asInstanceOf[DFDLTime]
+  override def getDateTime: DFDLDateTime = v.asInstanceOf[DFDLDateTime]
+  override def getByteArray: Array[Byte] = v.asInstanceOf[Array[Byte]]
+  override def getBoolean: JBoolean = v.asInstanceOf[JBoolean]
+  override def getNumber: JNumber = v.asInstanceOf[JNumber]
+  override def getByte: JByte = v.asInstanceOf[JByte]
+  override def getShort: JShort = v.asInstanceOf[JShort]
+  override def getInt: JInt = v.asInstanceOf[JInt]
+  override def getLong: JLong = v.asInstanceOf[JLong]
+  override def getDouble: JDouble = v.asInstanceOf[JDouble]
+  override def getFloat: JFloat = v.asInstanceOf[JFloat]
+  override def getBigInt: JBigInt = v.asInstanceOf[JBigInt]
+  override def getString: JString = v.asInstanceOf[JString]
+  override def getURI: URI = v.asInstanceOf[URI]

Review Comment:
   Can these just call `dataValue.getXYZ`? Doesn't that already do the asInstanceOf stuff? Avoids some duplication and makes it so if we ever chang DataValue this gets those changes?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/XMLInfosetOutputterMixin.scala:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.infoset
+
+import org.apache.daffodil.lib.equality._
+import org.apache.daffodil.lib.util.Maybe
+import org.apache.daffodil.lib.xml.XMLUtils
+
+trait XMLInfosetOutputterMixin {
+
+  def remapped(dataValueAsString: String) =
+    XMLUtils.remapXMLIllegalCharactersToPUA(dataValueAsString)
+
+  /**
+   * String suitable for use in the text of a Processing Instruction.
+   *
+   * The text is a pseudo-XML string.
+   */
+  protected final def fmtInfo(diTerm: DITerm): Maybe[String] = {

Review Comment:
   Should we just drop the fmtInfo stuff? Seems like nothing uses it, and my guess it's is only used for debugging? If we do need access to fmtInfo, we should make that available in the CLI or VS Code debugger.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/RuntimeData.scala:
##########
@@ -689,6 +698,10 @@ sealed class ElementRuntimeData(
       name
     }
   }
+
+  override def namespace: NS = dpathElementCompileInfo.namedQName.namespace
+  override def optNamespacePrefix: Option[String] = dpathElementCompileInfo.namedQName.prefix
+  override def sscd: String = dpathElementCompileInfo.sscd

Review Comment:
   I've forgotten what does sscd stands for. Which makes me thinks it's used rarely enough that maybe it needs a more descriptive name? Could be done in another PR, since it's not related to this.



##########
daffodil-japi/src/main/scala/org/apache/daffodil/japi/infoset/Infoset.scala:
##########
@@ -161,7 +160,7 @@ abstract class InfosetOutputter extends SInfosetOutputter {
    */
 
   @throws[Exception]
-  def startSimple(diSimple: DISimple): Unit
+  def startSimple(diSimple: InfosetSimpleElement): Unit

Review Comment:
   We should rename the parameter names too, probably just drop the `di`, e.g. `diSimple` -> `simple`.



##########
daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/QNameBase.scala:
##########
@@ -350,6 +349,27 @@ trait QNameBase extends Serializable {
     if (prefix.isDefined) prefix.get + ":" + local else local
   }
 
+  private lazy val _Prefix_LocalName = toQNameString.replace(":", "_")
+
+  /**
+   * Maps the QName to a name like prefix_localName for use with
+   * data models/representations that don't have namespaces.
+   */
+  def toPrefix_localName = {

Review Comment:
   I'm not sure I love this name. I feel like this wants to be `toSafeString` or something like that. `safe` is probably the wrong word, but `toFooString` feels like a more clear name about what it's doing.



##########
daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/QNameBase.scala:
##########
@@ -350,6 +349,27 @@ trait QNameBase extends Serializable {
     if (prefix.isDefined) prefix.get + ":" + local else local
   }
 
+  private lazy val _Prefix_LocalName = toQNameString.replace(":", "_")

Review Comment:
   This logic feels like it wants to be in the MetaDataHandler or InfosetWalkers. Different implementations are possibly going to have different restrictions on what makes a valid name or a name that fits best into their system. I also imagine most implementations could get away with just dropping the namespace altogether. Schemas that rely on namespaces for uniqueness are fairly rare, and those that do exist should probably be changed or deprecated.



##########
daffodil-japi/src/main/scala/org/apache/daffodil/japi/Daffodil.scala:
##########
@@ -514,6 +508,13 @@ class DataProcessor private[japi] (private var dp: SDataProcessor)
    */
   def save(output: WritableByteChannel): Unit = dp.save(output)
 
+  /**
+   * Walks the handler over the runtime metadata structures
+   *
+   * @param handler - the handler is called-back during the walk as each metadata structure is encountered.
+   */
+  def walkMetadata(handler: MetadataHandler) = dp.walkMetadata(handler)

Review Comment:
   I'm not sure I love the name `walkMetadata`. If you're not familiar with Daffodil internals, I'm not sure you would know what "metadata" means and if you would want to walk it. That said, I can't think of a better name, so maybe it just needs a comment explaining what what it means to walk metadata.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement
+  def isNilled: Boolean
+  def isEmpty: Boolean

Review Comment:
   What does `isEmpty` represent? Seems an API user should never be handed an infoset element that is empty.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement
+  def isNilled: Boolean
+  def isEmpty: Boolean
+  def asSimple: InfosetSimpleElement
+  def asComplex: InfosetComplexElement
+
+  /*
+   * For API users to avoid the need to access RuntimeData
+   */
+  def metadata: ElementMetadata
+  def name = metadata.name
+  def namespace = metadata.namespace

Review Comment:
   Thoughts on dropping name and namespace since users have access to the metadata?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,80 @@
+package org.apache.daffodil.runtime1.api
+
+import org.apache.daffodil.lib.exceptions.SchemaFileLocation
+import org.apache.daffodil.lib.xml.NS
+import org.apache.daffodil.lib.xml.NamedQName
+import org.apache.daffodil.runtime1.dpath.PrimTypeNode
+
+/*
+ * This is the supportable API for access to the RuntimeData structures
+ */
+trait Metadata {
+  def schemaFileLocation: SchemaFileLocation

Review Comment:
   `SchemaFileLocation` isn't part of the api. In the JAPI/SAPI we map things to a `LocationInSchemaFile` (not sure if that's the same thing?). I'm not sure how this would show up in the JavaDocs



##########
daffodil-lib/src/main/scala/org/apache/daffodil/lib/xml/QNameBase.scala:
##########
@@ -350,6 +349,27 @@ trait QNameBase extends Serializable {
     if (prefix.isDefined) prefix.get + ":" + local else local
   }
 
+  private lazy val _Prefix_LocalName = toQNameString.replace(":", "_")

Review Comment:
   Can this just be something like `prefix + "_" + localName`?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetImpl.scala:
##########
@@ -1650,32 +1708,71 @@ sealed class DIComplex(override val erd: ElementRuntimeData)
     }
   }
 
+  final def getChild(
+    childMetadata: ElementMetadata,
+    enableLinearSearch: Boolean,

Review Comment:
   I suggest this goes away. Users should only get access to children via a walker. If they need by-name access, then they need to walk the infoset and build their own internal representation that gives them that.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement
+  def isNilled: Boolean
+  def isEmpty: Boolean
+  def asSimple: InfosetSimpleElement
+  def asComplex: InfosetComplexElement
+
+  /*
+   * For API users to avoid the need to access RuntimeData
+   */
+  def metadata: ElementMetadata
+  def name = metadata.name
+  def namespace = metadata.namespace
+}
+
+trait InfosetComplexElement extends InfosetElement {
+  override def metadata: ComplexElementMetadata
+  def getChildElement(childMetadata: ElementMetadata): InfosetElement
+  def getChildArray(childMetadata: ElementMetadata): InfosetArray
+  def getChild(childMetadata: ElementMetadata): InfosetItem
+}
+
+trait InfosetSimpleElement extends InfosetElement {
+
+  override def metadata: SimpleElementMetadata
+
+  /**
+   * Caches the string so we're not allocating strings repeatedly
+   */
+  def dataValueAsString: String
+
+  /*
+   * These are so that API users don't have to know about our
+   * very Scala-oriented DataValue type system.
+   */
+  def getAnyRef: AnyRef

Review Comment:
   Should we remove `getAnyRef`? A API user should never need that. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement
+  def isNilled: Boolean
+  def isEmpty: Boolean
+  def asSimple: InfosetSimpleElement
+  def asComplex: InfosetComplexElement
+
+  /*
+   * For API users to avoid the need to access RuntimeData
+   */
+  def metadata: ElementMetadata
+  def name = metadata.name
+  def namespace = metadata.namespace
+}
+
+trait InfosetComplexElement extends InfosetElement {
+  override def metadata: ComplexElementMetadata
+  def getChildElement(childMetadata: ElementMetadata): InfosetElement
+  def getChildArray(childMetadata: ElementMetadata): InfosetArray
+  def getChild(childMetadata: ElementMetadata): InfosetItem
+}
+
+trait InfosetSimpleElement extends InfosetElement {
+
+  override def metadata: SimpleElementMetadata
+
+  /**
+   * Caches the string so we're not allocating strings repeatedly
+   */
+  def dataValueAsString: String
+
+  /*
+   * These are so that API users don't have to know about our
+   * very Scala-oriented DataValue type system.
+   */
+  def getAnyRef: AnyRef
+  def getBigDecimal: JBigDecimal
+  def getCalendar: DFDLCalendar
+  def getDate: DFDLDate
+  def getTime: DFDLTime
+  def getDateTime: DFDLDateTime
+  def getByteArray: Array[Byte]
+  def getBoolean: JBoolean
+  def getNumber: JNumber
+  def getByte: JByte
+  def getShort: JShort
+  def getInt: JInt
+  def getLong: JLong
+  def getDouble: JDouble
+  def getFloat: JFloat
+  def getBigInt: JBigInt
+  def getString: JString
+  def getURI: URI
+}

Review Comment:
   We probably need javadocs on these functions individually. It's going to be repetetive, but is needed for Javadocs to look correct.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement
+  def isNilled: Boolean
+  def isEmpty: Boolean
+  def asSimple: InfosetSimpleElement
+  def asComplex: InfosetComplexElement
+
+  /*
+   * For API users to avoid the need to access RuntimeData
+   */
+  def metadata: ElementMetadata
+  def name = metadata.name
+  def namespace = metadata.namespace
+}
+
+trait InfosetComplexElement extends InfosetElement {
+  override def metadata: ComplexElementMetadata
+  def getChildElement(childMetadata: ElementMetadata): InfosetElement
+  def getChildArray(childMetadata: ElementMetadata): InfosetArray
+  def getChild(childMetadata: ElementMetadata): InfosetItem
+}
+
+trait InfosetSimpleElement extends InfosetElement {
+
+  override def metadata: SimpleElementMetadata
+
+  /**
+   * Caches the string so we're not allocating strings repeatedly
+   */
+  def dataValueAsString: String
+
+  /*
+   * These are so that API users don't have to know about our
+   * very Scala-oriented DataValue type system.
+   */
+  def getAnyRef: AnyRef
+  def getBigDecimal: JBigDecimal
+  def getCalendar: DFDLCalendar
+  def getDate: DFDLDate
+  def getTime: DFDLTime
+  def getDateTime: DFDLDateTime
+  def getByteArray: Array[Byte]
+  def getBoolean: JBoolean
+  def getNumber: JNumber
+  def getByte: JByte
+  def getShort: JShort
+  def getInt: JInt
+  def getLong: JLong
+  def getDouble: JDouble
+  def getFloat: JFloat
+  def getBigInt: JBigInt
+  def getString: JString
+  def getURI: URI
+}
+
+trait InfosetDocument extends InfosetItem {
+  override def metadata: ElementMetadata
+}
+
+trait InfosetItem {
+
+  def metadata: Metadata
+
+  /**
+   * The totalElementCount is the total count of how many elements this InfosetItem contains.
+   *
+   * (Used to call this 'size', but size is often a length-like thing, so changed name
+   * to be more distinctive)
+   */
+  def totalElementCount: Long

Review Comment:
   Do API users need this? I'm not even sure what this value represents or what it's used for. Looking at the implementation, totalElementCount is the number of children recursively? Seems like that number wouldn't be all that useful. And it might not even be accurate, since I imagine it's possible all the children haven't been finalized yet, since the infoset is often in a state where parts can be walked but parts can't.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement
+  def isNilled: Boolean
+  def isEmpty: Boolean
+  def asSimple: InfosetSimpleElement
+  def asComplex: InfosetComplexElement
+
+  /*
+   * For API users to avoid the need to access RuntimeData
+   */
+  def metadata: ElementMetadata
+  def name = metadata.name
+  def namespace = metadata.namespace
+}
+
+trait InfosetComplexElement extends InfosetElement {
+  override def metadata: ComplexElementMetadata
+  def getChildElement(childMetadata: ElementMetadata): InfosetElement
+  def getChildArray(childMetadata: ElementMetadata): InfosetArray
+  def getChild(childMetadata: ElementMetadata): InfosetItem
+}

Review Comment:
   Should we grant access to children this way? This allows users to walk and infoset themselves, which is dangerous. An infoset is often in a state where it is only partially built and children should not be accessed, which only Daffodil internals know when that is.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,80 @@
+package org.apache.daffodil.runtime1.api
+
+import org.apache.daffodil.lib.exceptions.SchemaFileLocation
+import org.apache.daffodil.lib.xml.NS
+import org.apache.daffodil.lib.xml.NamedQName
+import org.apache.daffodil.runtime1.dpath.PrimTypeNode
+
+/*
+ * This is the supportable API for access to the RuntimeData structures
+ */
+trait Metadata {
+  def schemaFileLocation: SchemaFileLocation
+  def path: String
+  def diagnosticDebugName: String
+  override def toString = diagnosticDebugName
+}
+
+trait TermMetadata extends Metadata {
+  def isArray: Boolean
+}
+
+trait ElementMetadata extends TermMetadata {
+  def isSimpleType: Boolean
+  def isComplexType: Boolean
+
+  def name: String
+  def namespace: NS
+  def optNamespacePrefix: Option[String]
+  def isArray: Boolean
+  def isOptional: Boolean
+  def namedQName: NamedQName
+
+  def sscd: String
+  def isNillable: Boolean
+
+  def runtimeProperties: java.util.Map[String, String]
+
+}
+
+trait ComplexElementMetadata extends ElementMetadata {
+
+  def childMetadata: Seq[ElementMetadata]

Review Comment:
   Similar questions as earlier, should we grant access to children stuff? That would allow users to implement their own walkers, which feels unsafe. Seems we only want to give access to this element and only this element. They'll see the childMetadata when we walk to it.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement
+  def isNilled: Boolean
+  def isEmpty: Boolean
+  def asSimple: InfosetSimpleElement
+  def asComplex: InfosetComplexElement
+
+  /*
+   * For API users to avoid the need to access RuntimeData
+   */
+  def metadata: ElementMetadata
+  def name = metadata.name
+  def namespace = metadata.namespace
+}
+
+trait InfosetComplexElement extends InfosetElement {
+  override def metadata: ComplexElementMetadata
+  def getChildElement(childMetadata: ElementMetadata): InfosetElement
+  def getChildArray(childMetadata: ElementMetadata): InfosetArray
+  def getChild(childMetadata: ElementMetadata): InfosetItem
+}
+
+trait InfosetSimpleElement extends InfosetElement {
+
+  override def metadata: SimpleElementMetadata
+
+  /**
+   * Caches the string so we're not allocating strings repeatedly
+   */
+  def dataValueAsString: String
+
+  /*
+   * These are so that API users don't have to know about our
+   * very Scala-oriented DataValue type system.
+   */
+  def getAnyRef: AnyRef
+  def getBigDecimal: JBigDecimal
+  def getCalendar: DFDLCalendar
+  def getDate: DFDLDate
+  def getTime: DFDLTime
+  def getDateTime: DFDLDateTime

Review Comment:
   These calendar/time types are kindof internal I think? We might want to think if there's something better to return here, like a `Calendar` or some other Java date/time structure.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetImpl.scala:
##########
@@ -1157,6 +1180,8 @@ final class DIArray(
 ) extends DINode
   with InfosetArray {
 
+  override def metadata: ElementMetadata = erd.asInstanceOf[ElementMetadata]

Review Comment:
   Is `asInstanceOf` needed here? If `erd` implements `ElementMetaData`, should it be fine to just return erd?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetImpl.scala:
##########
@@ -1079,27 +1101,27 @@ sealed trait DIElement
   protected final var _isHidden = false
   final def isHidden: Boolean = _isHidden
 
-  override def setHidden(): Unit = {
+  def setHidden(): Unit = {
     _isHidden = true
   }
 
   final def runtimeData = erd
-  protected final var _parent: InfosetComplexElement = null
+  protected final var _parent: DIComplex = null

Review Comment:
   This is a great change and something we should do everywhere infosets appear in the code (maybe this PR does that?). Seems Daffodil wants use the `DI` variant internally everywhere. The only place the non-`DI` variant should is the public functions like the walkers so we can limit what parts a user has acess to. But Daffodil internals should have access to all parts of a DI. I feel like there are a number of places where that's not the case (though maybe this PR fixed all those?) and should make a lot of `asInstanceOf[DIFoo]` go away.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,80 @@
+package org.apache.daffodil.runtime1.api
+
+import org.apache.daffodil.lib.exceptions.SchemaFileLocation
+import org.apache.daffodil.lib.xml.NS
+import org.apache.daffodil.lib.xml.NamedQName
+import org.apache.daffodil.runtime1.dpath.PrimTypeNode
+
+/*
+ * This is the supportable API for access to the RuntimeData structures
+ */
+trait Metadata {
+  def schemaFileLocation: SchemaFileLocation
+  def path: String
+  def diagnosticDebugName: String
+  override def toString = diagnosticDebugName
+}
+
+trait TermMetadata extends Metadata {
+  def isArray: Boolean
+}
+
+trait ElementMetadata extends TermMetadata {
+  def isSimpleType: Boolean
+  def isComplexType: Boolean
+
+  def name: String
+  def namespace: NS
+  def optNamespacePrefix: Option[String]
+  def isArray: Boolean
+  def isOptional: Boolean
+  def namedQName: NamedQName
+
+  def sscd: String

Review Comment:
   Need a new name for sscd if this is going to be publicly available to users.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/ScalaXMLInfosetOutputter.scala:
##########
@@ -17,20 +17,23 @@
 
 package org.apache.daffodil.runtime1.infoset
 
-import scala.collection.mutable.ListBuffer
-import scala.xml.MetaData
-import scala.xml.Null
-import scala.xml.UnprefixedAttribute
-
 import org.apache.daffodil.lib.exceptions.Assert
 import org.apache.daffodil.lib.util.MStackOf
 import org.apache.daffodil.lib.util.Maybe
 import org.apache.daffodil.lib.xml.XMLUtils
+import org.apache.daffodil.runtime1.api.InfosetArray
+import org.apache.daffodil.runtime1.api.InfosetComplexElement
+import org.apache.daffodil.runtime1.api.InfosetSimpleElement
 import org.apache.daffodil.runtime1.dpath.NodeInfo
 
+import scala.collection.mutable.ListBuffer
+import scala.xml.MetaData
+import scala.xml.Null
+import scala.xml.UnprefixedAttribute

Review Comment:
   You might need to change your IDE settings. I think scala imports are supposed to be before daffodil imports. Can you configure your IDE to use the .scalafmt.conf file?



##########
daffodil-japi/src/main/scala/org/apache/daffodil/japi/Daffodil.scala:
##########
@@ -514,6 +508,13 @@ class DataProcessor private[japi] (private var dp: SDataProcessor)
    */
   def save(output: WritableByteChannel): Unit = dp.save(output)
 
+  /**
+   * Walks the handler over the runtime metadata structures
+   *
+   * @param handler - the handler is called-back during the walk as each metadata structure is encountered.
+   */
+  def walkMetadata(handler: MetadataHandler) = dp.walkMetadata(handler)

Review Comment:
   Very good points. Suggest these be added to a comment somewhere.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/JDOMInfosetOutputter.scala:
##########
@@ -46,7 +49,8 @@ class JDOMInfosetOutputter extends InfosetOutputter with XMLInfosetOutputter {
     result = Maybe(root.asInstanceOf[org.jdom2.Document])
   }
 
-  def startSimple(diSimple: DISimple): Unit = {
+  override def startSimple(se: InfosetSimpleElement): Unit = {
+    val diSimple = se.asInstanceOf[DISimple]

Review Comment:
   Is casting to a DISimple done to avoid having to update other outputters and deal with that later? I assume the goal is these infoset outputters should only use InfosetFooEelements? Doing the conversion would be good evidence that the InfosetElement API interface makes everything needed available.



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/MetadataWalker.scala:
##########
@@ -0,0 +1,76 @@
+package org.apache.daffodil.runtime1.processors
+
+import org.apache.daffodil.lib.exceptions.Assert
+import org.apache.daffodil.runtime1.api.MetadataHandler
+import org.apache.daffodil.runtime1.api.SequenceMetadata
+
+/**
+ * Walks the schema, but not the DSOM schema, it walks the RuntimeData objects that
+ * represent the DFDL schema at runtime.
+ *
+ * @param dp
+ */
+class MetadataWalker(private val dp: DataProcessor) {
+
+  private lazy val rootERD = dp.ssrd.elementRuntimeData
+
+  def walk(handler: MetadataHandler): Unit = {
+    walkTerm(handler, rootERD)
+  }
+
+  private def walkTerm(handler: MetadataHandler, trd: TermRuntimeData): Unit = {
+    trd match {
+      case err: ErrorERD => Assert.invariantFailed("should not get ErrorERDs")
+      case erd: ElementRuntimeData => walkElement(handler, erd)
+      case srd: SequenceRuntimeData => walkSequence(handler, srd)
+      case crd: ChoiceRuntimeData => walkChoice(handler, crd)
+      case _ => Assert.invariantFailed(s"unrecognized TermRuntimeData subtype: $trd")
+    }
+  }
+
+  private def walkElement(handler: MetadataHandler, erd: ElementRuntimeData): Unit = {
+    if (erd.optComplexTypeModelGroupRuntimeData.isDefined)
+      walkComplexElement(handler, erd)
+    else
+      walkSimpleElement(handler, erd)
+  }
+
+  private def walkComplexElement(
+    handler: MetadataHandler,
+    erd: ElementRuntimeData,
+  ): Unit = {
+    val mgrd = erd.optComplexTypeModelGroupRuntimeData.getOrElse {
+      Assert.invariantFailed("not a complex type element")
+    }
+    handler.startComplexElementMetadata(erd)
+    walkTerm(handler, mgrd)
+    handler.endComplexElementMetadata(erd)
+  }
+
+  private def walkSimpleElement(
+    handler: MetadataHandler,
+    erd: ElementRuntimeData,
+  ): Unit = {
+    handler.simpleElementMetadata(erd)
+  }
+
+  private def walkSequence(handler: MetadataHandler, sm: SequenceMetadata): Unit = {
+    val srd = sm.asInstanceOf[SequenceRuntimeData]
+    if (!srd.isHidden) {
+      // handler.startSequence()
+      srd.groupMembers.map { trd =>
+        walkTerm(handler, trd)
+      }
+      // handler.endSequence()
+    }
+  }
+
+  private def walkChoice(handler: MetadataHandler, crd: ChoiceRuntimeData): Unit = {
+    // handler.startChoice()

Review Comment:
   Seems a walker handler would want to know that the children are options of a choice and could be converted to a union or something. Can they know that without start/endChoice?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on code in PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#discussion_r1388432873


##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement
+  def isNilled: Boolean
+  def isEmpty: Boolean
+  def asSimple: InfosetSimpleElement
+  def asComplex: InfosetComplexElement
+
+  /*
+   * For API users to avoid the need to access RuntimeData
+   */
+  def metadata: ElementMetadata
+  def name = metadata.name
+  def namespace = metadata.namespace

Review Comment:
   These are just about having to change fewer lines in the tests. But I'll see how impactful this is. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,80 @@
+package org.apache.daffodil.runtime1.api
+
+import org.apache.daffodil.lib.exceptions.SchemaFileLocation
+import org.apache.daffodil.lib.xml.NS
+import org.apache.daffodil.lib.xml.NamedQName
+import org.apache.daffodil.runtime1.dpath.PrimTypeNode
+
+/*
+ * This is the supportable API for access to the RuntimeData structures
+ */
+trait Metadata {
+  def schemaFileLocation: SchemaFileLocation

Review Comment:
   We do need this, because say the Apache Drill integration can't handle if multiple children of an element have the same name. We need to issue a diagnostic and complain about the DFDL schema appropriately. 
   
   SchemaFileLocation is just a little tuple, so the easiest thing is just to give access to its members from this class. That way it doesn't shine through as an object. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/JDOMInfosetOutputter.scala:
##########
@@ -46,7 +49,8 @@ class JDOMInfosetOutputter extends InfosetOutputter with XMLInfosetOutputter {
     result = Maybe(root.asInstanceOf[org.jdom2.Document])
   }
 
-  def startSimple(diSimple: DISimple): Unit = {
+  override def startSimple(se: InfosetSimpleElement): Unit = {
+    val diSimple = se.asInstanceOf[DISimple]

Review Comment:
   Yes, but, there is some use of internals due to issues with namespace binding minimization in the XML outputters. 
   Some complex logic there that I'm not inclined to try to refactor currently. 
   
   I have changed most usage of DINode classes to InfosetFooElements. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,80 @@
+package org.apache.daffodil.runtime1.api
+
+import org.apache.daffodil.lib.exceptions.SchemaFileLocation
+import org.apache.daffodil.lib.xml.NS
+import org.apache.daffodil.lib.xml.NamedQName
+import org.apache.daffodil.runtime1.dpath.PrimTypeNode
+
+/*
+ * This is the supportable API for access to the RuntimeData structures
+ */
+trait Metadata {
+  def schemaFileLocation: SchemaFileLocation
+  def path: String
+  def diagnosticDebugName: String
+  override def toString = diagnosticDebugName
+}
+
+trait TermMetadata extends Metadata {
+  def isArray: Boolean
+}
+
+trait ElementMetadata extends TermMetadata {
+  def isSimpleType: Boolean
+  def isComplexType: Boolean
+
+  def name: String
+  def namespace: NS
+  def optNamespacePrefix: Option[String]
+  def isArray: Boolean
+  def isOptional: Boolean
+  def namedQName: NamedQName
+
+  def sscd: String
+  def isNillable: Boolean
+
+  def runtimeProperties: java.util.Map[String, String]
+
+}
+
+trait ComplexElementMetadata extends ElementMetadata {
+
+  def childMetadata: Seq[ElementMetadata]
+}
+
+trait SimpleElementMetadata extends ElementMetadata {
+  def primType: PrimTypeNode
+}
+
+trait ModelGroupMetadata extends TermMetadata {}
+
+trait SequenceMetadata extends ModelGroupMetadata {}
+
+trait ChoiceMetadata extends ModelGroupMetadata {}
+
+/**
+ * Base class used by clients who want to walk the runtime schema information.
+ */
+abstract class MetadataHandler() {
+
+  /**
+   * Called for simple type element metadata (for declarations or references)
+   */
+  def simpleElementMetadata(m: SimpleElementMetadata): Unit
+
+  /**
+   * Called for complex type element metadata (for declarations or references)
+   *
+   * Subsequent calls will be for the model group making up the content
+   * of the element.
+   */
+  def startComplexElementMetadata(m: ComplexElementMetadata): Unit
+  def endComplexElementMetadata(m: ComplexElementMetadata): Unit
+
+  //  def startSequenceMetadata(m: SequenceMetadata): Unit

Review Comment:
   Note: Test that these are actually walked and not just parent direct to element children. 
   There are use cases for these involving the API user needing some sort of NextElementResolver so they can cope with DFDL schemas where not all children have unique names. 



##########
daffodil-japi/src/main/scala/org/apache/daffodil/japi/Daffodil.scala:
##########
@@ -401,6 +388,13 @@ class DataProcessor private[japi] (private var dp: SDataProcessor)
   extends WithDiagnostics(dp)
   with Serializable {
 
+  /**
+   * For use by the JAPI implementation-internals only.
+   *
+   * @return the underlying data processor
+   */
+  private[japi] def getUnderlyingDataProcessor = dp

Review Comment:
   Remove if unused. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement
+  def isNilled: Boolean
+  def isEmpty: Boolean
+  def asSimple: InfosetSimpleElement
+  def asComplex: InfosetComplexElement
+
+  /*
+   * For API users to avoid the need to access RuntimeData
+   */
+  def metadata: ElementMetadata
+  def name = metadata.name
+  def namespace = metadata.namespace
+}
+
+trait InfosetComplexElement extends InfosetElement {
+  override def metadata: ComplexElementMetadata
+  def getChildElement(childMetadata: ElementMetadata): InfosetElement
+  def getChildArray(childMetadata: ElementMetadata): InfosetArray
+  def getChild(childMetadata: ElementMetadata): InfosetItem
+}
+
+trait InfosetSimpleElement extends InfosetElement {
+
+  override def metadata: SimpleElementMetadata
+
+  /**
+   * Caches the string so we're not allocating strings repeatedly
+   */
+  def dataValueAsString: String
+
+  /*
+   * These are so that API users don't have to know about our
+   * very Scala-oriented DataValue type system.
+   */
+  def getAnyRef: AnyRef
+  def getBigDecimal: JBigDecimal
+  def getCalendar: DFDLCalendar
+  def getDate: DFDLDate
+  def getTime: DFDLTime
+  def getDateTime: DFDLDateTime

Review Comment:
   I think they are just encapsulations around java types for some serialization related reason. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement

Review Comment:
   You points on this and access to children are well taken. For metadata, walking that stuff is fine. For data, we don't want people crawling the infoset themselves. 
   
   If these are needed by tests I'll  make them package private. If not then I'll remove them. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement
+  def isNilled: Boolean
+  def isEmpty: Boolean
+  def asSimple: InfosetSimpleElement
+  def asComplex: InfosetComplexElement
+
+  /*
+   * For API users to avoid the need to access RuntimeData
+   */
+  def metadata: ElementMetadata
+  def name = metadata.name
+  def namespace = metadata.namespace
+}
+
+trait InfosetComplexElement extends InfosetElement {
+  override def metadata: ComplexElementMetadata
+  def getChildElement(childMetadata: ElementMetadata): InfosetElement
+  def getChildArray(childMetadata: ElementMetadata): InfosetArray
+  def getChild(childMetadata: ElementMetadata): InfosetItem
+}
+
+trait InfosetSimpleElement extends InfosetElement {
+
+  override def metadata: SimpleElementMetadata
+
+  /**
+   * Caches the string so we're not allocating strings repeatedly
+   */
+  def dataValueAsString: String
+
+  /*
+   * These are so that API users don't have to know about our
+   * very Scala-oriented DataValue type system.
+   */
+  def getAnyRef: AnyRef

Review Comment:
   Actually, in the Drill changes I've done getAnyRef is the only thing I've called in these. Turns out I need an object, not a primitive value, to give it to Drill. Calling getAnyRef is easier since the assignment to the field takes an object ref.
   
   I'd almost suggest we leave out all the typed getters and ONLY have getAnyRef. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,80 @@
+package org.apache.daffodil.runtime1.api
+
+import org.apache.daffodil.lib.exceptions.SchemaFileLocation
+import org.apache.daffodil.lib.xml.NS
+import org.apache.daffodil.lib.xml.NamedQName
+import org.apache.daffodil.runtime1.dpath.PrimTypeNode
+
+/*
+ * This is the supportable API for access to the RuntimeData structures
+ */
+trait Metadata {
+  def schemaFileLocation: SchemaFileLocation
+  def path: String
+  def diagnosticDebugName: String
+  override def toString = diagnosticDebugName
+}
+
+trait TermMetadata extends Metadata {
+  def isArray: Boolean
+}
+
+trait ElementMetadata extends TermMetadata {
+  def isSimpleType: Boolean
+  def isComplexType: Boolean
+
+  def name: String
+  def namespace: NS
+  def optNamespacePrefix: Option[String]
+  def isArray: Boolean
+  def isOptional: Boolean
+  def namedQName: NamedQName
+
+  def sscd: String

Review Comment:
   Unused, so I'll pull it. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement
+  def isNilled: Boolean
+  def isEmpty: Boolean
+  def asSimple: InfosetSimpleElement
+  def asComplex: InfosetComplexElement

Review Comment:
   These are for tests to make them more compact. Removing. They are seldom used. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)
+}
+
+/**
+ * API access to elements of the DFDL Infoset
+ */
+trait InfosetElement extends InfosetItem {
+  def parent: InfosetComplexElement
+  def isNilled: Boolean
+  def isEmpty: Boolean

Review Comment:
   empty string or empty hexBinary. Will add the scaladoc. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/XMLInfosetOutputterMixin.scala:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.infoset
+
+import org.apache.daffodil.lib.equality._
+import org.apache.daffodil.lib.util.Maybe
+import org.apache.daffodil.lib.xml.XMLUtils
+
+trait XMLInfosetOutputterMixin {
+
+  def remapped(dataValueAsString: String) =
+    XMLUtils.remapXMLIllegalCharactersToPUA(dataValueAsString)
+
+  /**
+   * String suitable for use in the text of a Processing Instruction.
+   *
+   * The text is a pseudo-XML string.
+   */
+  protected final def fmtInfo(diTerm: DITerm): Maybe[String] = {

Review Comment:
   Dropped. It was unused. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetImpl.scala:
##########
@@ -1079,27 +1101,27 @@ sealed trait DIElement
   protected final var _isHidden = false
   final def isHidden: Boolean = _isHidden
 
-  override def setHidden(): Unit = {
+  def setHidden(): Unit = {
     _isHidden = true
   }
 
   final def runtimeData = erd
-  protected final var _parent: InfosetComplexElement = null
+  protected final var _parent: DIComplex = null

Review Comment:
   So one thing this change to expose the InfosetItem means.... users can just downcast and they have the DINode objects again. 
   
   The proxy way of isolating an API is superior in that one can't even get a pointer to the interior object, so you can't bypass the API. 
   
   On the other hand, the jars are all linkable, and so people who bypass the API can easily just get at whatever they want. Furthermore it's all open source..... 
   
   A way of avoiding exposing the InfosetNode traits/classes at all is to make the walkers work more like a cursor. 
   I.e., startComplex() and other event-handler methods don't get passed anything. Just the methods that access the data and metadata fields are all on the handler (in this case the InfosetOutputter) and can be called to access the underlying info for the current object the handler is positioned on. 
   
   But I don't want to do that simply because it's yet more proxy-style duplication of API. It's also unexpected stylistically I think. 
   
   
   



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,80 @@
+package org.apache.daffodil.runtime1.api
+
+import org.apache.daffodil.lib.exceptions.SchemaFileLocation
+import org.apache.daffodil.lib.xml.NS
+import org.apache.daffodil.lib.xml.NamedQName
+import org.apache.daffodil.runtime1.dpath.PrimTypeNode
+
+/*
+ * This is the supportable API for access to the RuntimeData structures
+ */
+trait Metadata {
+  def schemaFileLocation: SchemaFileLocation
+  def path: String
+  def diagnosticDebugName: String
+  override def toString = diagnosticDebugName
+}
+
+trait TermMetadata extends Metadata {
+  def isArray: Boolean
+}
+
+trait ElementMetadata extends TermMetadata {
+  def isSimpleType: Boolean
+  def isComplexType: Boolean
+
+  def name: String
+  def namespace: NS
+  def optNamespacePrefix: Option[String]
+  def isArray: Boolean
+  def isOptional: Boolean
+  def namedQName: NamedQName
+
+  def sscd: String
+  def isNillable: Boolean
+
+  def runtimeProperties: java.util.Map[String, String]
+
+}
+
+trait ComplexElementMetadata extends ElementMetadata {
+
+  def childMetadata: Seq[ElementMetadata]

Review Comment:
   Actually, the whole metadata structure is static, so there is no risk to this. I do have need of it in the Apache Drill integration in one place where we want to know if the docRoot has exactly 1 complex repeating child element, and if so we treat the docRoot differently. 



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/infoset/InfosetImpl.scala:
##########
@@ -1558,6 +1590,29 @@ sealed class DISimple(override val erd: ElementRuntimeData)
     Assert.invariantFailed("Should not try to remove a child of a simple type")
   }
 
+  // call to getAnyRef here seems redundant
+  // but tests fail that compare objects if this call isn't here
+  // Boxing/Unboxing related perhaps
+  @inline private def v = dataValue.getAnyRef
+
+  override def getAnyRef: AnyRef = v.asInstanceOf[AnyRef]
+  override def getBigDecimal: JBigDecimal = v.asInstanceOf[JBigDecimal]
+  override def getCalendar: DFDLCalendar = v.asInstanceOf[DFDLCalendar]
+  override def getDate: DFDLDate = v.asInstanceOf[DFDLDate]
+  override def getTime: DFDLTime = v.asInstanceOf[DFDLTime]
+  override def getDateTime: DFDLDateTime = v.asInstanceOf[DFDLDateTime]
+  override def getByteArray: Array[Byte] = v.asInstanceOf[Array[Byte]]
+  override def getBoolean: JBoolean = v.asInstanceOf[JBoolean]
+  override def getNumber: JNumber = v.asInstanceOf[JNumber]
+  override def getByte: JByte = v.asInstanceOf[JByte]
+  override def getShort: JShort = v.asInstanceOf[JShort]
+  override def getInt: JInt = v.asInstanceOf[JInt]
+  override def getLong: JLong = v.asInstanceOf[JLong]
+  override def getDouble: JDouble = v.asInstanceOf[JDouble]
+  override def getFloat: JFloat = v.asInstanceOf[JFloat]
+  override def getBigInt: JBigInt = v.asInstanceOf[JBigInt]
+  override def getString: JString = v.asInstanceOf[JString]
+  override def getURI: URI = v.asInstanceOf[URI]

Review Comment:
   I tried to do it that way, but invoking these from Java was having trouble. I'll put one back and see what happens though, and document why it has to be done this way (if that holds). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle merged PR #1112:
URL: https://github.com/apache/daffodil/pull/1112


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on code in PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#discussion_r1390161343


##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Infoset.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Boolean => JBoolean }
+import java.lang.{ Byte => JByte }
+import java.lang.{ Double => JDouble }
+import java.lang.{ Float => JFloat }
+import java.lang.{ Integer => JInt }
+import java.lang.{ Long => JLong }
+import java.lang.{ Number => JNumber }
+import java.lang.{ Short => JShort }
+import java.lang.{ String => JString }
+import java.math.{ BigDecimal => JBigDecimal }
+import java.math.{ BigInteger => JBigInt }
+import java.net.URI
+
+import org.apache.daffodil.lib.calendar.DFDLCalendar
+import org.apache.daffodil.lib.calendar.DFDLDate
+import org.apache.daffodil.lib.calendar.DFDLDateTime
+import org.apache.daffodil.lib.calendar.DFDLTime
+
+/**
+ * API access to array objects in the DFDL Infoset
+ */
+trait InfosetArray extends InfosetItem {
+  override def metadata: ElementMetadata // the element that is an array
+  def getOccurrence(occursIndex: Long): InfosetElement
+  def length: Long
+  final def apply(occursIndex: Long): InfosetElement = getOccurrence(occursIndex)
+  final def apply(occursIndex: Int) : InfosetElement = getOccurrence(occursIndex.toLong)

Review Comment:
   I will do this last. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on code in PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#discussion_r1390333828


##########
daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/SequenceGroup.scala:
##########
@@ -70,6 +70,11 @@ abstract class SequenceTermBase(
 
   def isOrdered: Boolean
 
+  /**
+   * Overridden in sequence group ref
+   */
+  def isHidden: Boolean = false

Review Comment:
   This requires a test to ensure that if the same group definition is used twice, one from a dfdl:hiddenGroupRef, the other from a regular group ref, that Daffodil recognizes this and makes two disjoint copies. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#issuecomment-1810344662

   Not intended for that use case.
   
   This new metadata walker is for java (typically) applications to interface
   to runtime1 specifically.
   
   The c backend is more of a compiler integration.
   
   On Tue, Nov 14, 2023, 9:06 AM John Interrante ***@***.***>
   wrote:
   
   > @stevedlawrence <https://github.com/stevedlawrence> @tuxji
   > <https://github.com/tuxji> This should be very close to ready now. Please
   > give this a review again.
   >
   > I'll start reviewing this PR. Quick question, though. Is this new walker
   > API intended to be used by the C code generator too, instead of the C
   > generator walking the compiled schema by hand?
   >
   > —
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/daffodil/pull/1112#issuecomment-1810272164>,
   > or unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AALUDA2STEIZ6GIJEIZOX53YEN3G5AVCNFSM6AAAAAA7D4HOC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJQGI3TEMJWGQ>
   > .
   > You are receiving this because you authored the thread.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["tuxji (via GitHub)" <gi...@apache.org>]

tuxji commented on code in PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#discussion_r1392700059


##########
daffodil-core/src/test/scala/org/apache/daffodil/core/infoset/TestInfoset.scala:
##########
@@ -183,16 +191,18 @@ class TestInfoset1 {
     val (infoset: DIComplex, _, tunable) = testInfoset(testSchema, xmlInfoset)
     val Seq(w_erd) = infoset.erd.childERDs
     infoset.getChildArray(w_erd, tunable) match {
-      case arr: DIArray => {
+      case arr: InfosetArray => {
         assertEquals(2, arr.length)
-        var a = arr(1).asInstanceOf[InfosetSimpleElement]
-        assertEquals(w_erd, a.runtimeData)
+        var a = arr(1).asInstanceOf[InfosetSimpleElement] // 1-based
+        assertEquals(w_erd, a.metadata)
 
-        assertEquals(4, a.dataValue.getAnyRef)
-        assertEquals(infoset, a.parent)
+        assertEquals(4, a.getUnsignedLong.longValue().toInt)

Review Comment:
   What is the reason for calling `.longValue()` when it seems possible to call `a.getUnsignedLong.toInt` directly?



##########
build.sbt:
##########
@@ -219,7 +219,7 @@ val minSupportedJavaVersion: String =
 
 lazy val commonSettings = Seq(
   organization := "org.apache.daffodil",
-  version := "3.6.0",
+  version := "3.7.0-SNAPSHOT",

Review Comment:
   This change doesn't belong in this PR.  PR #1113 which hasn't been merged yet will make this change for us.



##########
daffodil-japi/src/test/java/org/apache/daffodil/example/TestJavaAPI.java:
##########
@@ -1209,7 +1209,6 @@ public void testJavaAPI25() throws IOException, ClassNotFoundException, External
         assertEquals(expectedData, bos.toString());
     }
 
-    @Test

Review Comment:
   Is this change intentional and why is the test being disabled?



##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.runtime1.api
+
+import java.lang.{ Long => JLong }
+import scala.collection.JavaConverters._
+import scala.xml.NamespaceBinding
+
+import org.apache.daffodil.runtime1.dpath.NodeInfo
+import org.apache.daffodil.runtime1.dpath.NodeInfo.PrimType
+
+/**
+ * This is the supportable API for access to the RuntimeData structures
+ * which provide access to static information about a given .

Review Comment:
   About a given compiled schema object, or is there a better description than that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["mbeckerle (via GitHub)" <gi...@apache.org>]

mbeckerle commented on code in PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#discussion_r1390120034


##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/api/Metadata.scala:
##########
@@ -0,0 +1,80 @@
+package org.apache.daffodil.runtime1.api
+
+import org.apache.daffodil.lib.exceptions.SchemaFileLocation
+import org.apache.daffodil.lib.xml.NS
+import org.apache.daffodil.lib.xml.NamedQName
+import org.apache.daffodil.runtime1.dpath.PrimTypeNode
+
+/*
+ * This is the supportable API for access to the RuntimeData structures
+ */
+trait Metadata {
+  def schemaFileLocation: SchemaFileLocation
+  def path: String
+  def diagnosticDebugName: String
+  override def toString = diagnosticDebugName
+}
+
+trait TermMetadata extends Metadata {
+  def isArray: Boolean
+}
+
+trait ElementMetadata extends TermMetadata {
+  def isSimpleType: Boolean
+  def isComplexType: Boolean
+
+  def name: String
+  def namespace: NS
+  def optNamespacePrefix: Option[String]
+  def isArray: Boolean
+  def isOptional: Boolean
+  def namedQName: NamedQName
+
+  def sscd: String
+  def isNillable: Boolean
+
+  def runtimeProperties: java.util.Map[String, String]
+
+}
+
+trait ComplexElementMetadata extends ElementMetadata {
+
+  def childMetadata: Seq[ElementMetadata]

Review Comment:
   Removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Draft: Refactoring to expose clean metadata and infoset walkers [daffodil]

["tuxji (via GitHub)" <gi...@apache.org>]

tuxji commented on PR #1112:
URL: https://github.com/apache/daffodil/pull/1112#issuecomment-1810272164

   > @stevedlawrence @tuxji This should be very close to ready now. Please give this a review again.
   
   I'll start reviewing this PR.  Quick question, though.  Is this new walker API intended to be used by the C code generator too, instead of the C generator walking the compiled schema by hand?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org