You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by GitBox <gi...@apache.org> on 2020/03/09 18:27:34 UTC

[GitHub] [incubator-daffodil] olabusayoT opened a new pull request #332: Refactor isHidden

olabusayoT opened a new pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332
 
 
   - Removes isHidden from ElementRuntimeData
   - Adds isHidden flag to ModelGroup/ModelGroupRuntimeData
   - Adds private hiddenDepth counter (and associated functions)
   increment/decrement logic (HiddenGroupCombinators) for PState/UState
   - overrides increment/decrement functions in UStateForSuspension to
   fail on UsageError as these have no place in that context
   - Adds check to verify  we are out of hidden state at the end of
   parsing/unparsing
   - Adds isHidden check to ChoiceGroupRef
   - Adds isHidden/setHidden logic to InfosetElement/DIElement
   - Adds tests for group defs used in both hidden/non-hidden sequence &
   choice contexts
   -- adds "info hidden" to debugger with tests
   -- moves InfoOccursIndex to maintain alphabetic ordering of objects
   
   DAFFODIL-2281

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r389892751
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/ModelGroupGrammarMixin.scala
 ##########
 @@ -68,9 +69,16 @@ trait ModelGroupGrammarMixin
           case c: ChoiceTermBase => DelimiterStackCombinatorChoice(c, content)
           case s: SequenceTermBase => DelimiterStackCombinatorSequence(s, content)
         }
-      } else { groupContentDef }
+      } else {
+        groupContent
+      } //TODO: confirm this works; changed from groupContentDef
 
-    finalContent
+      val mgrd = self.modelGroupRuntimeData
+      if (mgrd.isHidden) {
 
 Review comment:
   Does isHidden need to be part of the MoelGroupRuntimeData? Can we just use isHidden frome the ModelGroup DSOM?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r392950118
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/SchemaComponent.scala
 ##########
 @@ -243,12 +227,12 @@ trait SchemaComponent
         case ct: ComplexTypeBase => "ct"
         case st: SimpleTypeDefBase => "st=" + st.namedQName.toQNameString
         case st: SimpleTypeBase => "st=" + st.primType.globalQName.toQNameString
-        case cgr: ChoiceGroupRef => "cgr" + (if (cgr.position > 1) cgr.position else "") + "=" + cgr.groupDef.namedQName.toQNameString
+        case cgr: ChoiceGroupRef => "cgr" + (if (cgr.isHidden) "h" else "") + (if (cgr.position > 1) cgr.position else "") + "=" + cgr.groupDef.namedQName.toQNameString
 
 Review comment:
   I noticed SequenceGroupRef below doesn't use ``toQNameString``. Is that a bug in SequenceGroupRef, or is there some fundamental differece between choices and groups? Maybe the toString function in namedQName just calls toQNameString and it's redundant?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r397994308
 
 

 ##########
 File path: daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/ExpressionEvaluatingUnparsers.scala
 ##########
 @@ -135,6 +135,9 @@ class TypeValueCalcUnparser(typeCalculator: TypeCalculator, repTypeUnparser: Unp
 
     val origInfosetElement = ustate.currentInfosetNode
     val tmpInfosetElement = Infoset.newElement(repTypeRuntimeData).asInstanceOf[DISimple]
+    if (ustate.withinHiddenNest)
 
 Review comment:
   It's possible you don't need this. These temp elements are transient for repType. But it doesn't hurt, and it is consistent that these be marked hidden. So I'd leave it in. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r390348529
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Term.scala
 ##########
 @@ -732,9 +728,8 @@ trait Term
   final def possibleFirstChildElementsInInfoset: Seq[ElementBase] = LV('possibleFirstChildElementsInInfoset) {
     val pfct = possibleFirstChildTerms
     val firstChildren = pfct.flatMap {
-      case e: ElementBase if e.isHidden => Nil
       case e: ElementBase => Seq(e)
-      case s: SequenceTermBase if s.isHidden => Nil
+      case gr: GroupRef if gr.asModelGroup.isHidden => Nil
 
 Review comment:
   Just curious how difficult it would be to move isHidden (or isHiddenGroupRef, agreed that's amore clear name) to GroupRef? Now that you mention it, that does feel like that's maybe a more natural place for this property.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r390360358
 
 

 ##########
 File path: daffodil-runtime1/src/main/scala/org/apache/daffodil/debugger/InteractiveDebugger.scala
 ##########
 @@ -1497,6 +1482,17 @@ class InteractiveDebugger(runner: InteractiveDebuggerRunner, eCompilers: Express
         }
       }
 
+      object InfoHidden extends DebugCommand with DebugCommandValidateZeroArgs {
+        val name = "hidden"
+        override lazy val short = "hid"
 
 Review comment:
   I think if you don't define ``short`` then it just defaults to the first letter of the name, which in this case is "h" and is a reasonable short form for ``hidden``. Nothing else starts with h.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] olabusayoT commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
olabusayoT commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r393015723
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/ChoiceCombinator.scala
 ##########
 @@ -217,27 +217,37 @@ case class ChoiceCombinator(ch: ChoiceTermBase, alternatives: Seq[Gram])
   }
 
   override lazy val unparser: Unparser = {
-    val optDefaultableBranchUnparser = if (ch.isHidden) {
-      // this call is necessary since it will throw an SDE if no choice branch
-      // was defaultable
-      ch.childrenThatAreRecursivelyNotDefaultableOrOVC
+    /*
+     * Since it's impossible to know the hiddenness for terms at this level (unless
+     * they're a hiddenGroupRef), we always attempt to find a defaultable unparser.
+     * If we find one, and at runtime, we're in a hidden state, we can use that. If we
+     * are not in a hidden state, it doesn't get used. If we don't find one, and we encounter
+     * a hidden group ref here, we know we'll run into that issue at runtime, so we SDE.
+     */
+    val optDefaultableBranchUnparser = {
+      val defaultableBranches = ch.groupMembers.find { _.isFullyDefaultableOrOVC }
+      if (defaultableBranches.nonEmpty) {
+        val defaultableBranchRD = defaultableBranches.get.runtimeData
 
-      val defaultableBranches = ch.groupMembers.filter { _.childrenThatAreRecursivelyNotDefaultableOrOVC.length == 0 }
-      Assert.invariant(defaultableBranches.length > 0)
-      if (ch.isHidden && defaultableBranches.length > 1) {
-        SDW(WarnID.ChoiceInsideHiddenGroup, "xs:choice inside a hidden group has unparse ambiguity: multiple branches exist with all children either defaulable or have the dfdl:outputValueCalc property set. The first branch will be chosen during unparse. Defaultable branches are:\n%s",
-          defaultableBranches.mkString("\n"))
+        // Choices inside a hidden group ref are slightly different because we
+        // will never see events for any of the branches. Instead, we will just
+        // always pick the branch in which every thing is defaultble or OVC.
+        val defaultableBranchUnparser = alternatives.find(_.context.runtimeData =:= defaultableBranchRD).get.unparser
+        Some(defaultableBranchUnparser)
+      } else {
+        ch match {
+          case cgr: GroupRef if cgr.isHidden => {
+            /*
+             * This is a requirement for terms inside a hidden group. If we're a hidden group ref, we know
+             * our descendants need this requirements, so it's a SDE if they don't meet it.
 
 Review comment:
   I meant "require this", it's been update to clarify

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r397997231
 
 

 ##########
 File path: daffodil-runtime1/src/main/scala/org/apache/daffodil/infoset/PartialNextElementResolver.scala
 ##########
 @@ -76,12 +76,8 @@ trait NextElementResolver { self: InfosetInputter =>
         // set of possible elements.
         //
         maybeERD = resolver.maybeNextElement(name, nameSpace, hasNamespace)
-        //
-        // If the ERD is marked hidden, then we never expect any events corresponding
-        // to its elements.
-        //
-        if (maybeERD.isDefined && maybeERD.get.isHidden)
-          maybeERD = Nope // clobber it back to Nope
+        // we address the cases where ERD is in a hidden context where the elements are assigned
 
 Review comment:
   Rephrase. "The cases where .... are addressed where..." 
   I had a hard time following the comment in the current wording. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r389896732
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/runtime1/ChoiceTermRuntime1Mixin.scala
 ##########
 @@ -140,6 +140,7 @@ trait ChoiceTermRuntime1Mixin { self: ChoiceTermBase =>
       optIgnoreCase,
       maybeFillByteEv,
       maybeCheckByteAndBitOrderEv,
-      maybeCheckBitOrderAndCharset)
+      maybeCheckBitOrderAndCharset,
+      isHidden)
 
 Review comment:
   Is this needed? Seems all hidden logic is either done at compile time or using the new hidden stack thing.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r389898011
 
 

 ##########
 File path: daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/HiddenGroupCombinatorUnparser.scala
 ##########
 @@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.processors.unparsers
+
+import org.apache.daffodil.processors.ModelGroupRuntimeData
+
+/**
+ * The purpose of this combinator is to increment/decrement the depth counter
+ * for hidden groups, and call the group's unparser. As we get deeper into the
+ * hiddenGroupRefs, we'll increment the counter, call the group's unparser, and as
+ * we unwind from the refs, we'll decrement.
+ */
+class HiddenGroupCombinatorUnparser(
+  ctxt: ModelGroupRuntimeData, bodyUnparser: Unparser)
+  extends CombinatorUnparser(ctxt) {
+
+  override lazy val childProcessors = Vector(bodyUnparser)
+
+  override lazy val runtimeDependencies = Vector()
+
+  def unparse(start: UState): Unit = {
+    start.incrementHiddenDef
+
+    // unparse
+    bodyUnparser.unparse1(start)
+
+    start.decrementHiddenDef
+  }
 
 Review comment:
   We probably want to wrap this in a a try/finally so that the hiddenef count always decremented even if the bodyUnparse throws some exception. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r389898693
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/runtime1/SequenceTermRuntime1Mixin.scala
 ##########
 @@ -76,7 +77,8 @@ trait ChoiceBranchImpliedSequenceRuntime1Mixin { self: ChoiceBranchImpliedSequen
       None,
       Maybe.Nope,
       Maybe.Nope,
-      Maybe.Nope)
+      Maybe.Nope,
+      isHidden)
 
 Review comment:
   This should always be false here. I.e., a ChoiceBranchImpliedSequence cannot be created for a group ref. So you can write
   ```
   {
   Assert.invariant(!isHidden)
   false
   }
   ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r389890072
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/ModelGroupGrammarMixin.scala
 ##########
 @@ -68,9 +69,16 @@ trait ModelGroupGrammarMixin
           case c: ChoiceTermBase => DelimiterStackCombinatorChoice(c, content)
           case s: SequenceTermBase => DelimiterStackCombinatorSequence(s, content)
         }
-      } else { groupContentDef }
+      } else {
+        groupContent
+      } //TODO: confirm this works; changed from groupContentDef
 
 Review comment:
   Can we remove this TODO? I think we know this works now.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r389899042
 
 

 ##########
 File path: daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/parsers/HiddenGroupCombinatorParser.scala
 ##########
 @@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.processors.parsers
+
+import org.apache.daffodil.processors.ModelGroupRuntimeData
+
+/**
+ * The purpose of this combinator is to increment/decrement the depth counter
+ * for hidden groups, call the group's parser. As we get deeper into the
+ * hiddenGroupRefs, we'll increment the counter, call the group's parser, and as
+ * we unwind from the refs, we'll decrement.
+ */
+class HiddenGroupCombinatorParser(
+  ctxt: ModelGroupRuntimeData, bodyParser: Parser)
+  extends CombinatorParser(ctxt) {
+
+  override lazy val childProcessors = Vector(bodyParser)
+
+  override lazy val runtimeDependencies = Vector()
+
+  def parse(start: PState): Unit = {
+    start.incrementHiddenDef
+
+    // parse
+    bodyParser.parse1(start)
+
+    start.decrementHiddenDef
 
 Review comment:
   Same here, probably need a try/finally.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] olabusayoT commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
olabusayoT commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r393827299
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/ChoiceCombinator.scala
 ##########
 @@ -257,9 +267,8 @@ case class ChoiceCombinator(ch: ChoiceTermBase, alternatives: Seq[Gram])
     if (mapValues.isEmpty)
       new NadaUnparser(null)
     else {
-      val firstBranchUnparser = alternatives(0).unparser
       new ChoiceCombinatorUnparser(ch.modelGroupRuntimeData, eventUnparserMap,
-        choiceLengthInBits, optDefaultableBranchUnparser, firstBranchUnparser)
+        choiceLengthInBits, optDefaultableBranchUnparser)
 
 Review comment:
   Both have been tested and are passing

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r397978972
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/GroupRef.scala
 ##########
 @@ -129,18 +131,15 @@ final class ChoiceGroupRef(
   refXML: Node,
   refLexicalParent: SchemaComponent,
   positionArg: Int,
-  isHiddenArg: Boolean)
+  override val isHidden: Boolean)
   extends ChoiceTermBase(refXML, Option(refLexicalParent), positionArg)
   with GroupRef {
 
   requiredEvaluationsIfActivated(groupDef)
 
   private lazy val globalGroupDef = globalGroupDefArg // once only
 
-  override def isHidden = isHiddenArg
-  override def isHiddenGroupRef = isHiddenArg
-
-  override lazy val groupDef = globalGroupDef
+  override lazy val groupDef = LV('cgrGroupDef){ globalGroupDef }.value
 
 Review comment:
   Is it important for the symbol used in the LV match the variable name, maybe for debugging or diagnostic purposes?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r390413928
 
 

 ##########
 File path: daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/ProcessorStateBases.scala
 ##########
 @@ -480,6 +480,17 @@ abstract class ParseOrUnparseState protected (
     res
   }
 
+  private var _hiddenDepth = 0
+
+  def incrementHiddenDef = {
+    _hiddenDepth += 1
+  }
+  def decrementHiddenDef = {
+    _hiddenDepth -= 1
+  }
+
+  def withinHiddenNest: Boolean = _hiddenDepth > 0
 
 Review comment:
   That would be good to do. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r397987021
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/GrammarTerm.scala
 ##########
 @@ -94,16 +94,19 @@ abstract class Gram(contextArg: SchemaComponent)
    * Note: This should not evaluate the argument unless it has to.
    */
   def ~(qq: => Gram) = {
-    lazy val q = qq.deref
+    val q = qq.deref
     val self = this.deref
 
     val res =
       if (self.isEmpty) {
-        if (q.isEmpty)
+        if (q.isEmpty) {
           EmptyGram
-        else q
-      } else if (q.isEmpty) self
-      else {
+        } else {
+          q
+        }
+      } else if (q.isEmpty) {
+        self
 
 Review comment:
   I think this is the only change here? I'm not sure I understandt this change. What issue does this fix?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] olabusayoT commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
olabusayoT commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r398040388
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Term.scala
 ##########
 @@ -578,14 +563,13 @@ trait Term
   protected def possibleFirstChildTerms: Seq[Term]
 
   /*
-     * Returns list of Elements that could be the first child in the infoset of this model group or element.
-     */
+   * Returns list of Elements that could be the first child in the infoset of this model group or element.
+   */
   final lazy val possibleFirstChildElementsInInfoset: Seq[ElementBase] = LV('possibleFirstChildElementsInInfoset) {
 
 Review comment:
   I've created DAFFODIL-2305 to clean up these elements and some of the tests can't be cleanly deleted, so this can be merged separately.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r397999636
 
 

 ##########
 File path: daffodil-test/src/test/resources/org/apache/daffodil/section14/sequence_groups/SequenceGroup.tdml
 ##########
 @@ -1406,9 +1406,9 @@
       <tdml:documentPart type="text"><![CDATA[145,2]]></tdml:documentPart>
     </tdml:document>
     <tdml:errors>
-      <tdml:error>choice</tdml:error>
-      <tdml:error>hidden group</tdml:error>
-      <tdml:error>outputValueCalc</tdml:error>
+      <tdml:error>Schema Definition Error</tdml:error>
+      <tdml:error>hidden choice</tdml:error>
+      <tdml:error>must be fully defaultable or define dfdl:outputValueCalc</tdml:error>
 
 Review comment:
   Better to be less specific about the error messages so that minor changes to them may not break the test.
   I would just have the error checks look for "defaultable" and "outputValueCalc" somewhere in the message. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r392943983
 
 

 ##########
 File path: daffodil-cli/src/it/scala/org/apache/daffodil/debugger/TestCLIDebugger.scala
 ##########
 @@ -900,10 +900,14 @@ class TestCLIdebugger {
   }
 
   @Test def test_CLI_Debugger_InfoHidden_1() {
-    val schemaFile = Util.daffodilPath("daffodil-test/src/test/resources/org/apache/daffodil/section14/sequence_groups/SequencesWithHiddenRefs.dfdl.xsd")
-    val inputFile = Util.daffodilPath("daffodil-cli/src/it/resources/org/apache/daffodil/CLI/input/test_isHidden1.txt")
-    val (testSchemaFile, testInputFile) = if (Util.isWindows) (Util.cmdConvert(schemaFile), Util.cmdConvert(inputFile)) else (schemaFile, inputFile)
-
+    val schemaFile = Util.daffodilPath(
 
 Review comment:
   Agreed, I thought we had a bug for this already, but I can't find it. I've created [DAFFODIL-2292](https://issues.apache.org/jira/browse/DAFFODIL-2292) to track this issue. We probably shouldn't attempt ot fix it as part of this bug.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] olabusayoT commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
olabusayoT commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r390352687
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Term.scala
 ##########
 @@ -746,7 +741,7 @@ trait Term
    *
    * What stops this is when the end of an enclosing element has to be next.
    */
-  final def possibleNextChildElementsInInfoset: Seq[ElementBase] = LV('possibleNextChildElementsInInfoset) {
+  def possibleNextChildElementsInInfoset: Seq[ElementBase] = LV('possibleNextChildElementsInInfoset) {
 
 Review comment:
   I think this was an oversight. I'd changed it to a val during debugging, and forgot the "final" during restoration

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r390371278
 
 

 ##########
 File path: daffodil-test/src/test/resources/org/apache/daffodil/section14/sequence_groups/SequencesWithHiddenRefs.dfdl.xsd
 ##########
 @@ -0,0 +1,94 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+  xmlns:fn="http://www.w3.org/2005/xpath-functions"
+  xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:ex="http://example.com"
+  targetNamespace="http://example.com" elementFormDefault="unqualified">
+  <xs:include
+    schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
+  <xs:annotation>
+    <xs:appinfo source="http://www.ogf.org/dfdl/">
+      <dfdl:format ref="ex:GeneralFormat" lengthKind="delimited" />
+    </xs:appinfo>
+  </xs:annotation>
+  <xs:group name="s1">
+    <xs:sequence>
+      <xs:element name="f" type="xs:int" dfdl:outputValueCalc="{ 1 }" />
+    </xs:sequence>
+  </xs:group>
+  <xs:element name="e1" dfdl:lengthKind="delimited">
+    <xs:complexType>
+      <xs:sequence dfdl:separator=",">
+        <xs:sequence dfdl:hiddenGroupRef="ex:s1" />
+        <xs:element name="g" type="xs:int" />
+      </xs:sequence>
+    </xs:complexType>
+  </xs:element>
+  <xs:element name="e2" dfdl:lengthKind="delimited">
+    <xs:complexType>
+      <xs:sequence dfdl:separator=",">
+        <xs:group ref="ex:s1" />
+        <xs:element name="g" type="xs:int" />
+      </xs:sequence>
+    </xs:complexType>
+  </xs:element>
+  <xs:element name="e3" dfdl:lengthKind="delimited">
+    <xs:complexType>
+      <xs:sequence dfdl:separator=",">
+        <xs:sequence dfdl:hiddenGroupRef="ex:s1" />
+        <xs:element name="g">
+          <xs:complexType>
+            <xs:group ref="ex:s1" />
+          </xs:complexType>
+        </xs:element>
+      </xs:sequence>
+    </xs:complexType>
+  </xs:element>
+  <xs:group name="s2">
+    <xs:sequence>
+      <xs:element name="f">
+        <xs:complexType>
+          <xs:sequence>
+            <xs:element name="h">
+              <xs:complexType>
+                <xs:group ref="ex:s1" />
+              </xs:complexType>
+            </xs:element>
+          </xs:sequence>
+        </xs:complexType>
+      </xs:element>
+    </xs:sequence>
+  </xs:group>
+  <xs:element name="e4" dfdl:lengthKind="delimited">
+    <xs:complexType>
+      <xs:sequence dfdl:separator="~">
+        <xs:sequence dfdl:hiddenGroupRef="ex:s2" />
+        <xs:group ref="ex:s2" />
+      </xs:sequence>
+    </xs:complexType>
+  </xs:element>
+  <xs:element name="e5" dfdl:lengthKind="delimited">
+    <xs:complexType>
+      <xs:sequence dfdl:separator="~">
+        <xs:group ref="ex:s2" />
+        <xs:sequence dfdl:hiddenGroupRef="ex:s2" />
+      </xs:sequence>
+    </xs:complexType>
+  </xs:element>
+</xs:schema>
 
 Review comment:
   It might also be useful to have a test where a hidden group ref contains another hidden group ref just to ensure that nesting works as expected.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r389894439
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/ChoiceCombinator.scala
 ##########
 @@ -217,46 +217,49 @@ case class ChoiceCombinator(ch: ChoiceTermBase, alternatives: Seq[Gram])
   }
 
   override lazy val unparser: Unparser = {
-    if (!ch.isHidden) {
-      val eventRDMap = ch.choiceBranchMap
-      val eventUnparserMap = eventRDMap.flatMap {
-        case (cbe, rd) =>
-          // if we don't find a matching RD for a term that's probably
-          // because the term is an empty sequence or empty choice (which do happen
-          // and we even have tests for them). Since those can never be chosen by
-          // means of an element event, they don't appear in the map.
-          val altGram = alternatives.find { alt =>
-            val crd = alt.context.runtimeData
-            val found = crd =:= rd
-            found
-          }
-          altGram.map { ag => (cbe, ag.unparser) }
-      }
-      val mapValues = eventUnparserMap.map { case (k, v) => v }.toSeq.filterNot(_.isEmpty)
-      if (mapValues.isEmpty)
-        new NadaUnparser(null)
-      else {
-        new ChoiceCombinatorUnparser(ch.modelGroupRuntimeData, eventUnparserMap, choiceLengthInBits)
-      }
-    } else {
-      // Choices inside a hidden group ref are slightly different because we
-      // will never see events for any of the branches. Instead, we will just
-      // always pick the branch in which every thing is defaultble or OVC. It
-      // is a warning if more than one of those branches exist. It is an SDE if
-      // such a branch does not exist, which is detected elsewhere
-
+    val optDefaultableBranchUnparser = if (ch.isHidden) {
 
 Review comment:
   Does this logic need to be in the runtime unparsers? We can only know if a choice is hidden if it is directly referenced by a hidden group ref. But if a choice is a child of a hidden group ref then this can't know if it's hidden or not.
   
   Do we need to just always determine a defaultable branch and pass that into the choice combiantor, and then only use that if we determine we are hidden at runtime?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r391066153
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/ModelGroupGrammarMixin.scala
 ##########
 @@ -68,9 +69,16 @@ trait ModelGroupGrammarMixin
           case c: ChoiceTermBase => DelimiterStackCombinatorChoice(c, content)
           case s: SequenceTermBase => DelimiterStackCombinatorSequence(s, content)
         }
-      } else { groupContentDef }
+      } else {
+        groupContent
+      } //TODO: confirm this works; changed from groupContentDef
 
-    finalContent
+      val mgrd = self.modelGroupRuntimeData
+      if (mgrd.isHidden) {
 
 Review comment:
   :+1: 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r392961023
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/ChoiceCombinator.scala
 ##########
 @@ -217,46 +217,58 @@ case class ChoiceCombinator(ch: ChoiceTermBase, alternatives: Seq[Gram])
   }
 
   override lazy val unparser: Unparser = {
-    if (!ch.isHidden) {
-      val eventRDMap = ch.choiceBranchMap
-      val eventUnparserMap = eventRDMap.flatMap {
-        case (cbe, rd) =>
-          // if we don't find a matching RD for a term that's probably
-          // because the term is an empty sequence or empty choice (which do happen
-          // and we even have tests for them). Since those can never be chosen by
-          // means of an element event, they don't appear in the map.
-          val altGram = alternatives.find { alt =>
-            val crd = alt.context.runtimeData
-            val found = crd =:= rd
-            found
+    /*
+     * Since it's impossible to know the hiddenness for terms at this level (unless
+     * they're a hiddenGroupRef), we always attempt to find a defaultable unparser.
+     * If we find one, and at runtime, we're in a hidden state, we can use that. If we
+     * are not in a hidden state, it doesn't get used. If we don't find one, and we encounter
+     * a hidden group ref here, we know we'll run into that issue at runtime, so we SDE.
 
 Review comment:
   I think the last sentence of this comment isn't correct. Looks like the code below is correct in that we can always determine staticaly if we are a hidden group that doesn't contains a defaultable branch.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r397971675
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Term.scala
 ##########
 @@ -578,14 +563,13 @@ trait Term
   protected def possibleFirstChildTerms: Seq[Term]
 
   /*
-     * Returns list of Elements that could be the first child in the infoset of this model group or element.
-     */
+   * Returns list of Elements that could be the first child in the infoset of this model group or element.
+   */
   final lazy val possibleFirstChildElementsInInfoset: Seq[ElementBase] = LV('possibleFirstChildElementsInInfoset) {
 
 Review comment:
   This method seems to be unused and can be deleted. I thought it was deleted, but it does appear still on master. 
   
   The method couldBeLastElementInModelGroup, which has an abstract definition below, is also not used and it should be deleted. (The implementations of these are in ModelGroup and ElementBase.)
   
   Ditto for possibleFirstChildTerms above. 
   Ditto for isSequendceChild above (line 393)
   Ditto for possibleNextSiblingTerms below. 
   
   I can't put comments directly on the lines, but I walked through this file looking for unused methods.
   
   hasLaterRequiredSiblings is used only by unit tests, so that AND the unit tests looking at it can go away. 
   Same for hasPriorRequiredSiblings.
   
   priorSibling is unused and can be deleted. 
   
   Wow, that's quite a bit of code that can be deleted now. 
   (double check all these - I checked them on master. It's unlikely, but possible your PR has added a usage of one of these. IDEA's Find Usages is great.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r390396869
 
 

 ##########
 File path: daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/ProcessorStateBases.scala
 ##########
 @@ -480,6 +480,17 @@ abstract class ParseOrUnparseState protected (
     res
   }
 
+  private var _hiddenDepth = 0
+
+  def incrementHiddenDef = {
+    _hiddenDepth += 1
+  }
+  def decrementHiddenDef = {
+    _hiddenDepth -= 1
+  }
+
+  def withinHiddenNest: Boolean = _hiddenDepth > 0
 
 Review comment:
   The word "nest" here is important. The concept here is that the runtime is beneath/inside a hidden group, but not directly/statically, but anywhere within the dynamic context nest. 
   
   I'm open to other names, but the goal is to denote the dynamic extent of the hidden group. 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] olabusayoT commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
olabusayoT commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r398003835
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/GrammarTerm.scala
 ##########
 @@ -94,16 +94,19 @@ abstract class Gram(contextArg: SchemaComponent)
    * Note: This should not evaluate the argument unless it has to.
    */
   def ~(qq: => Gram) = {
-    lazy val q = qq.deref
+    val q = qq.deref
 
 Review comment:
   Ah that was an oversight, from when i was trying to debug the dropped HGR. Will fix

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r397986310
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/GrammarTerm.scala
 ##########
 @@ -94,16 +94,19 @@ abstract class Gram(contextArg: SchemaComponent)
    * Note: This should not evaluate the argument unless it has to.
    */
   def ~(qq: => Gram) = {
-    lazy val q = qq.deref
+    val q = qq.deref
 
 Review comment:
   The comment above says this shouldn't evaluate qq unless it needs to. But this looks like it will always evaluate it now. But maybe it always did that regardless?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r392956719
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Term.scala
 ##########
 @@ -612,46 +612,32 @@ trait Term
   }.value
 
   /*
-   * This function returns at list of simple elements that are descendents of
-   * this term that are not defaultable or OVC. This is a requirement for terms
-   * inside a hidden group. Note that there is an exception for choices, in
-   * which only a single branch be all defaultable or OVC. If any elements in a
-   * hidden group are not defaultable or OVC, then it is an SDE. This function
-   * assumes it is only called on elements inside of a hidden group.
+   * This function returns a boolean if the values of the term can be figured
+   * out during unparsing or if they don't need to appear in the infoset at all.
    *
-   * Note that this currently only requires OVC since default's aren't
-   * implemented. This function may need to change when we support defaults.
+   * Usually this means the term/its descendants have a default value (i.e defaultable),
+   * have defined dfdl:outputValueCalc, or are optional (minOccurs=0)
+   *
+   * Note that this currently only requires OVC and Optionally since defaults
+   * aren't fully implemented everywhere. This function may need to change when
+   * defaults are  fully implemented.
    */
-  lazy val childrenInHiddenGroupNotDefaultableOrOVC: Seq[ElementBase] = {
-    // this should only be called on hidden elements
-    val isH = isHidden
-    Assert.invariant(isH)
-
+  lazy val isFullyDefaultableOrOVC: Boolean = {
 
 Review comment:
   I"m wondering if we could come up with a different name since this also takes into account optionality but the name does not imply that? From your first sentence in the comment, maybe something like ``canUnparseIfHidden``. That way if the logic becomes more complex due to some new feature, we don't necessarily have to change the name.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r390366275
 
 

 ##########
 File path: daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/ProcessorStateBases.scala
 ##########
 @@ -480,6 +480,17 @@ abstract class ParseOrUnparseState protected (
     res
   }
 
+  private var _hiddenDepth = 0
+
+  def incrementHiddenDef = {
+    _hiddenDepth += 1
+  }
+  def decrementHiddenDef = {
+    _hiddenDepth -= 1
+  }
+
+  def withinHiddenNest: Boolean = _hiddenDepth > 0
 
 Review comment:
   Related to what Mike mentioned in the debugger, changing this variable to ``withinHiddenGroup`` is maybe more consistent with the DFDL spec.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] olabusayoT merged pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
olabusayoT merged pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r392523492
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/ChoiceCombinator.scala
 ##########
 @@ -217,27 +217,37 @@ case class ChoiceCombinator(ch: ChoiceTermBase, alternatives: Seq[Gram])
   }
 
   override lazy val unparser: Unparser = {
-    val optDefaultableBranchUnparser = if (ch.isHidden) {
-      // this call is necessary since it will throw an SDE if no choice branch
-      // was defaultable
-      ch.childrenThatAreRecursivelyNotDefaultableOrOVC
+    /*
+     * Since it's impossible to know the hiddenness for terms at this level (unless
+     * they're a hiddenGroupRef), we always attempt to find a defaultable unparser.
+     * If we find one, and at runtime, we're in a hidden state, we can use that. If we
+     * are not in a hidden state, it doesn't get used. If we don't find one, and we encounter
+     * a hidden group ref here, we know we'll run into that issue at runtime, so we SDE.
+     */
+    val optDefaultableBranchUnparser = {
+      val defaultableBranches = ch.groupMembers.find { _.isFullyDefaultableOrOVC }
+      if (defaultableBranches.nonEmpty) {
+        val defaultableBranchRD = defaultableBranches.get.runtimeData
 
-      val defaultableBranches = ch.groupMembers.filter { _.childrenThatAreRecursivelyNotDefaultableOrOVC.length == 0 }
-      Assert.invariant(defaultableBranches.length > 0)
-      if (ch.isHidden && defaultableBranches.length > 1) {
-        SDW(WarnID.ChoiceInsideHiddenGroup, "xs:choice inside a hidden group has unparse ambiguity: multiple branches exist with all children either defaulable or have the dfdl:outputValueCalc property set. The first branch will be chosen during unparse. Defaultable branches are:\n%s",
-          defaultableBranches.mkString("\n"))
+        // Choices inside a hidden group ref are slightly different because we
+        // will never see events for any of the branches. Instead, we will just
+        // always pick the branch in which every thing is defaultble or OVC.
+        val defaultableBranchUnparser = alternatives.find(_.context.runtimeData =:= defaultableBranchRD).get.unparser
+        Some(defaultableBranchUnparser)
+      } else {
+        ch match {
+          case cgr: GroupRef if cgr.isHidden => {
+            /*
+             * This is a requirement for terms inside a hidden group. If we're a hidden group ref, we know
+             * our descendants need this requirements, so it's a SDE if they don't meet it.
 
 Review comment:
   "need this requirements"... ambiguous. Did you mean "meets this requirement" or "requires this" ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] olabusayoT commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
olabusayoT commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r391088959
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/runtime1/SequenceTermRuntime1Mixin.scala
 ##########
 @@ -76,7 +77,8 @@ trait ChoiceBranchImpliedSequenceRuntime1Mixin { self: ChoiceBranchImpliedSequen
       None,
       Maybe.Nope,
       Maybe.Nope,
-      Maybe.Nope)
+      Maybe.Nope,
+      isHidden)
 
 Review comment:
   The Runtime structures no longer have an isHidden member

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r392946570
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/GroupRef.scala
 ##########
 @@ -30,6 +30,11 @@ trait GroupRef { self: ModelGroup =>
 
   final def referredToComponent = groupDef
 
+  /**
+   * Override in sequenceGroupRef and choiceGroupRef for hidden groups
+   */
+  def isHidden = false
 
 Review comment:
   Not a big deal at all, but more from a style perspective, I'm wondering if it makes sense to not provide a default value for this, so it's just ``def isHidden: Boolean``. The only two things that mixin ``GroupRef``s are ``SequenceGroupRef`` and ``ChoiceGroupRef`` and they both overridden this already. And if there's ever somthing else that mixes in ``GroupRef``, they'll be forced to think about how ``isHidden`` should be set or they'll get a compile error.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r389899025
 
 

 ##########
 File path: daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/ChoiceAndOtherVariousUnparsers.scala
 ##########
 @@ -29,69 +29,62 @@ import org.apache.daffodil.util.MaybeInt
 class ChoiceCombinatorUnparser(
   mgrd: ModelGroupRuntimeData,
   eventUnparserMap: Map[ChoiceBranchEvent, Unparser],
-  choiceLengthInBits: MaybeInt)
+  choiceLengthInBits: MaybeInt,
+  optDefaultableBranchUnparser: Option[Unparser],
+  firstBranchUnparser: Unparser)
   extends CombinatorUnparser(mgrd)
   with ToBriefXMLImpl {
   override def nom = "Choice"
 
   override lazy val runtimeDependencies = Vector()
 
-  override lazy val childProcessors = eventUnparserMap.map { case (k, v) => v }.toSeq.toVector
+  override lazy val childProcessors = (eventUnparserMap.map { case (k, v) => v }.toSeq ++ optDefaultableBranchUnparser).toVector
 
 Review comment:
   line too long. Please break at the ++

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r397988578
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/runtime1/ChoiceTermRuntime1Mixin.scala
 ##########
 @@ -96,20 +91,23 @@ trait ChoiceTermRuntime1Mixin { self: ChoiceTermBase =>
       // get recomputed, but it can appear with a stale value (left-over from parse)
       // in the arriving infoset for unparse.
       //
-      val optDefaultBranch = {
-
+      val optDefaultBranchIfHidden = {
 
 Review comment:
   Naming. The "ifHidden" suffix is misleading. This is the default branch, hidden or not. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r393887361
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/ChoiceCombinator.scala
 ##########
 @@ -257,9 +267,8 @@ case class ChoiceCombinator(ch: ChoiceTermBase, alternatives: Seq[Gram])
     if (mapValues.isEmpty)
       new NadaUnparser(null)
     else {
-      val firstBranchUnparser = alternatives(0).unparser
       new ChoiceCombinatorUnparser(ch.modelGroupRuntimeData, eventUnparserMap,
-        choiceLengthInBits, optDefaultableBranchUnparser, firstBranchUnparser)
+        choiceLengthInBits, optDefaultableBranchUnparser)
 
 Review comment:
   +1 then. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] olabusayoT commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
olabusayoT commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r391057073
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/ModelGroupGrammarMixin.scala
 ##########
 @@ -68,9 +69,16 @@ trait ModelGroupGrammarMixin
           case c: ChoiceTermBase => DelimiterStackCombinatorChoice(c, content)
           case s: SequenceTermBase => DelimiterStackCombinatorSequence(s, content)
         }
-      } else { groupContentDef }
+      } else {
+        groupContent
+      } //TODO: confirm this works; changed from groupContentDef
 
-    finalContent
+      val mgrd = self.modelGroupRuntimeData
+      if (mgrd.isHidden) {
 
 Review comment:
   After looking a tthe code, I realized we didn't actually need ModelGroupRuntimeData.isHidden, nor ModelGroup.isHidden (this was moved into GroupRef)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] olabusayoT commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
olabusayoT commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r391088444
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Term.scala
 ##########
 @@ -732,9 +728,8 @@ trait Term
   final def possibleFirstChildElementsInInfoset: Seq[ElementBase] = LV('possibleFirstChildElementsInInfoset) {
     val pfct = possibleFirstChildTerms
     val firstChildren = pfct.flatMap {
-      case e: ElementBase if e.isHidden => Nil
       case e: ElementBase => Seq(e)
-      case s: SequenceTermBase if s.isHidden => Nil
+      case gr: GroupRef if gr.asModelGroup.isHidden => Nil
 
 Review comment:
   I decided to move isHidden to GroupRef, as isHidden inherited from MG was only ever possibly true for these cases.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r392953986
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/SequenceGroup.scala
 ##########
 @@ -211,11 +211,16 @@ abstract class SequenceGroupTermBase(
   }
 
   lazy val checkHiddenSequenceIsDefaultableOrOVC: Unit = {
-    if (this.isHidden) {
 
 Review comment:
   I'm wondering if we can completely remove this function and replace it with something that is similar to what you've added in ChoiceCombinator to cause an SDE, but in SequenceCombinator?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] olabusayoT commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
olabusayoT commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r398003298
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/GroupRef.scala
 ##########
 @@ -129,18 +131,15 @@ final class ChoiceGroupRef(
   refXML: Node,
   refLexicalParent: SchemaComponent,
   positionArg: Int,
-  isHiddenArg: Boolean)
+  override val isHidden: Boolean)
   extends ChoiceTermBase(refXML, Option(refLexicalParent), positionArg)
   with GroupRef {
 
   requiredEvaluationsIfActivated(groupDef)
 
   private lazy val globalGroupDef = globalGroupDefArg // once only
 
-  override def isHidden = isHiddenArg
-  override def isHiddenGroupRef = isHiddenArg
-
-  override lazy val groupDef = globalGroupDef
+  override lazy val groupDef = LV('cgrGroupDef){ globalGroupDef }.value
 
 Review comment:
   There's a comment under LV, where it says 2 symbols with the same name could cause issues that tend to be difficult to debug, so I named it something different since SGR's groupDef already has a groupDef LV. I also found some precedence where the symbol didn't match the variable name (such as in org/apache/daffodil/dsom/SchemaDocIncludesAndImportsMixin.scala:151) so I figured it'd be ok

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r392961686
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/ChoiceCombinator.scala
 ##########
 @@ -217,46 +217,58 @@ case class ChoiceCombinator(ch: ChoiceTermBase, alternatives: Seq[Gram])
   }
 
   override lazy val unparser: Unparser = {
-    if (!ch.isHidden) {
-      val eventRDMap = ch.choiceBranchMap
-      val eventUnparserMap = eventRDMap.flatMap {
-        case (cbe, rd) =>
-          // if we don't find a matching RD for a term that's probably
-          // because the term is an empty sequence or empty choice (which do happen
-          // and we even have tests for them). Since those can never be chosen by
-          // means of an element event, they don't appear in the map.
-          val altGram = alternatives.find { alt =>
-            val crd = alt.context.runtimeData
-            val found = crd =:= rd
-            found
+    /*
+     * Since it's impossible to know the hiddenness for terms at this level (unless
+     * they're a hiddenGroupRef), we always attempt to find a defaultable unparser.
+     * If we find one, and at runtime, we're in a hidden state, we can use that. If we
+     * are not in a hidden state, it doesn't get used. If we don't find one, and we encounter
+     * a hidden group ref here, we know we'll run into that issue at runtime, so we SDE.
+     */
+    val optDefaultableBranchUnparser = {
 
 Review comment:
   Wondering if we could also change this name ,since ``defaultableBranchUnparser``doesn't really imply OVC-ability, or optionality. Maybsome something like ``unparseableBranchIfHidden``, or something similar to whatever name we come up with for isFullyDefaultableOrOVC?  

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r398093676
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Term.scala
 ##########
 @@ -578,14 +563,13 @@ trait Term
   protected def possibleFirstChildTerms: Seq[Term]
 
   /*
-     * Returns list of Elements that could be the first child in the infoset of this model group or element.
-     */
+   * Returns list of Elements that could be the first child in the infoset of this model group or element.
+   */
   final lazy val possibleFirstChildElementsInInfoset: Seq[ElementBase] = LV('possibleFirstChildElementsInInfoset) {
 
 Review comment:
   Good idea. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r397975275
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Term.scala
 ##########
 @@ -578,14 +563,13 @@ trait Term
   protected def possibleFirstChildTerms: Seq[Term]
 
 Review comment:
   Unused.  Can be deleted.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r390411621
 
 

 ##########
 File path: daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/ProcessorStateBases.scala
 ##########
 @@ -480,6 +480,17 @@ abstract class ParseOrUnparseState protected (
     res
   }
 
+  private var _hiddenDepth = 0
+
+  def incrementHiddenDef = {
+    _hiddenDepth += 1
+  }
+  def decrementHiddenDef = {
+    _hiddenDepth -= 1
+  }
+
+  def withinHiddenNest: Boolean = _hiddenDepth > 0
 
 Review comment:
   Fair enough. Should we modify the description of "info hidden" to mention "...within the nesting context of a hidden group" or something like that to be consistent, or is that distinction to verbose/excessive for the debugger?
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r392523694
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/ChoiceCombinator.scala
 ##########
 @@ -257,9 +267,8 @@ case class ChoiceCombinator(ch: ChoiceTermBase, alternatives: Seq[Gram])
     if (mapValues.isEmpty)
       new NadaUnparser(null)
     else {
-      val firstBranchUnparser = alternatives(0).unparser
       new ChoiceCombinatorUnparser(ch.modelGroupRuntimeData, eventUnparserMap,
-        choiceLengthInBits, optDefaultableBranchUnparser, firstBranchUnparser)
+        choiceLengthInBits, optDefaultableBranchUnparser)
 
 Review comment:
   Before committing this, we really must test mil-std-2045 and vmf. These have been turning up many problems related to choices for me. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r390363464
 
 

 ##########
 File path: daffodil-runtime1/src/main/scala/org/apache/daffodil/debugger/InteractiveDebugger.scala
 ##########
 @@ -1231,21 +1231,6 @@ class InteractiveDebugger(runner: InteractiveDebuggerRunner, eCompilers: Express
         }
       }
 
-      object InfoOccursIndex extends DebugCommand with DebugCommandValidateZeroArgs {
 
 Review comment:
   While we're reording things, it might be worth it to reorder the ``info`` subcommands Seq so they're alphabetical, and to make it so it's not on one ginormous line.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r389908595
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/ModelGroupGrammarMixin.scala
 ##########
 @@ -68,9 +69,16 @@ trait ModelGroupGrammarMixin
           case c: ChoiceTermBase => DelimiterStackCombinatorChoice(c, content)
           case s: SequenceTermBase => DelimiterStackCombinatorSequence(s, content)
         }
-      } else { groupContentDef }
+      } else {
+        groupContent
+      } //TODO: confirm this works; changed from groupContentDef
 
-    finalContent
+      val mgrd = self.modelGroupRuntimeData
+      if (mgrd.isHidden) {
 
 Review comment:
   Good point. This is the schema compiler, so we shouldn't be asking the runtime data objects, but the DSOM object. 
   The schema compiler creates the runtime data objects, but shouldn't use them. What they contain is in service of the runtime. 
   
   Interestingly, it's possible that ModelGroupRuntimeData doesn't need an isHidden member. Whether the parser/unparser is traversing a hidden group ref or not is represented by way of there being a HiddenGroupParser/Unparser constructed. So possibly we don't even need ModelGroupRuntimeData.isHidden. 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r389894066
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Term.scala
 ##########
 @@ -732,9 +728,8 @@ trait Term
   final def possibleFirstChildElementsInInfoset: Seq[ElementBase] = LV('possibleFirstChildElementsInInfoset) {
     val pfct = possibleFirstChildTerms
     val firstChildren = pfct.flatMap {
-      case e: ElementBase if e.isHidden => Nil
       case e: ElementBase => Seq(e)
-      case s: SequenceTermBase if s.isHidden => Nil
+      case gr: GroupRef if gr.asModelGroup.isHidden => Nil
 
 Review comment:
   The more I read this code the more I want to suggest changing this member name from isHidden to isHiddenGroupRef.
   
   That seems easier than the other clarifying change which would be to move isHidden onto GroupRef and remove it from ModelGroup.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r389901113
 
 

 ##########
 File path: daffodil-runtime1/src/main/scala/org/apache/daffodil/debugger/InteractiveDebugger.scala
 ##########
 @@ -1497,6 +1482,17 @@ class InteractiveDebugger(runner: InteractiveDebuggerRunner, eCompilers: Express
         }
       }
 
+      object InfoHidden extends DebugCommand with DebugCommandValidateZeroArgs {
+        val name = "hidden"
+        override lazy val short = "hid"
+        val desc = "display whether or not we're within a hidden state"
 
 Review comment:
   "... within a hidden group." would better match DFDL spec terminology.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r389896414
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/HiddenGroupCombinator.scala
 ##########
 @@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.grammar.primitives
+
+import org.apache.daffodil.dsom.ModelGroup
+import org.apache.daffodil.grammar.Gram
+import org.apache.daffodil.grammar.Terminal
+import org.apache.daffodil.processors.parsers.Parser
+import org.apache.daffodil.processors.unparsers.Unparser
+import org.apache.daffodil.processors.parsers.HiddenGroupCombinatorParser
+import org.apache.daffodil.processors.unparsers.HiddenGroupCombinatorUnparser
+
+final class HiddenGroupCombinator(
+    ctxt: ModelGroup,
+    body: Gram
+    ) extends Terminal(ctxt, !body.isEmpty){
+
+  lazy val parser: Parser = new HiddenGroupCombinatorParser( ctxt.modelGroupRuntimeData, body.parser)
+
+  override lazy val unparser: Unparser = new HiddenGroupCombinatorUnparser( ctxt.modelGroupRuntimeData, body.unparser)
+
+}
 
 Review comment:
   Check spaces here. There should be two-space indentation and there's some extra spaces.
   
    Also, the parenthesis closing the HiddenGroupCombinator should be on the same line as the last arguement. Eg.
   ```scala
   final class HiddenGroupCombinator(
     ctxt: ModelGroup,
     body: Gram)
     extends Terminal(ctxt, !body.isEmpty){
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] olabusayoT commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
olabusayoT commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r393086662
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/SchemaComponent.scala
 ##########
 @@ -243,12 +227,12 @@ trait SchemaComponent
         case ct: ComplexTypeBase => "ct"
         case st: SimpleTypeDefBase => "st=" + st.namedQName.toQNameString
         case st: SimpleTypeBase => "st=" + st.primType.globalQName.toQNameString
-        case cgr: ChoiceGroupRef => "cgr" + (if (cgr.position > 1) cgr.position else "") + "=" + cgr.groupDef.namedQName.toQNameString
+        case cgr: ChoiceGroupRef => "cgr" + (if (cgr.isHidden) "h" else "") + (if (cgr.position > 1) cgr.position else "") + "=" + cgr.groupDef.namedQName.toQNameString
 
 Review comment:
   So there is a difference b/w the toString and the toQNameString, the former considers the namespace in the case of no prefix, and will output '{ns}local' instead of the latter's 'local', but it doesn't look like there are any other differences beyond that. I'll update to 'toQNameString" so the same bhv is maintained across the cases.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r392964299
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/ChoiceCombinator.scala
 ##########
 @@ -217,46 +217,58 @@ case class ChoiceCombinator(ch: ChoiceTermBase, alternatives: Seq[Gram])
   }
 
   override lazy val unparser: Unparser = {
-    if (!ch.isHidden) {
-      val eventRDMap = ch.choiceBranchMap
-      val eventUnparserMap = eventRDMap.flatMap {
-        case (cbe, rd) =>
-          // if we don't find a matching RD for a term that's probably
-          // because the term is an empty sequence or empty choice (which do happen
-          // and we even have tests for them). Since those can never be chosen by
-          // means of an element event, they don't appear in the map.
-          val altGram = alternatives.find { alt =>
-            val crd = alt.context.runtimeData
-            val found = crd =:= rd
-            found
+    /*
+     * Since it's impossible to know the hiddenness for terms at this level (unless
+     * they're a hiddenGroupRef), we always attempt to find a defaultable unparser.
+     * If we find one, and at runtime, we're in a hidden state, we can use that. If we
+     * are not in a hidden state, it doesn't get used. If we don't find one, and we encounter
+     * a hidden group ref here, we know we'll run into that issue at runtime, so we SDE.
+     */
+    val optDefaultableBranchUnparser = {
+      val defaultableBranches = ch.groupMembers.find { _.isFullyDefaultableOrOVC }
+      if (defaultableBranches.nonEmpty) {
+        val defaultableBranchRD = defaultableBranches.get.runtimeData
+
+        // Choices inside a hidden group ref are slightly different because we
+        // will never see events for any of the branches. Instead, we will just
+        // always pick the branch in which every thing is defaultble or OVC.
+        val defaultableBranchUnparser = alternatives.find(_.context.runtimeData =:= defaultableBranchRD).get.unparser
+        Some(defaultableBranchUnparser)
+      } else {
+        ch match {
+          case cgr: GroupRef if cgr.isHidden => {
+            /*
+             * This is a requirement for terms inside a hidden group. If we're a hidden group ref, we know
+             * our descendants need this requirements, so it's a SDE if they don't meet it.
+             */
+            cgr.SDE("Element(s) of hidden group must be fully defaultable or define dfdl:outputValueCalc:\n%s",
 
 Review comment:
   This is a choice, so only one branch needs to be unparsable if hidden. The error message sortof implies all need to be unparsable. Also, this comment doesn't mention optionality, but maybe that's not necessary, since if someone hits this the solution is probabl to add a default or OVC.
   
   Also, if we got here, that means no branches were unparseable, so we can just print them all, e.g. ``ch.groupMembers.mkString("\n")``

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r392525136
 
 

 ##########
 File path: daffodil-cli/src/it/scala/org/apache/daffodil/debugger/TestCLIDebugger.scala
 ##########
 @@ -900,10 +900,14 @@ class TestCLIdebugger {
   }
 
   @Test def test_CLI_Debugger_InfoHidden_1() {
-    val schemaFile = Util.daffodilPath("daffodil-test/src/test/resources/org/apache/daffodil/section14/sequence_groups/SequencesWithHiddenRefs.dfdl.xsd")
-    val inputFile = Util.daffodilPath("daffodil-cli/src/it/resources/org/apache/daffodil/CLI/input/test_isHidden1.txt")
-    val (testSchemaFile, testInputFile) = if (Util.isWindows) (Util.cmdConvert(schemaFile), Util.cmdConvert(inputFile)) else (schemaFile, inputFile)
-
+    val schemaFile = Util.daffodilPath(
 
 Review comment:
   Really bugs me that every one of these tests (not just your new ones) has this repeated isWindows conditionalization. 
   
   We really should be able to abstract this away so that tests don't have this baggage over and over. 
   
   These same 6 lines of code are repeated a lot. 
   
   ```
       val (testSchemaFile, testInputFile) = if (Util.isWindows) {
         (Util.cmdConvert(schemaFile), Util.cmdConvert(inputFile.getAbsolutePath))
       } else {
         (schemaFile, inputFile)
      } 
        val shell = if (Util.isWindows) Util.start("", envp = DAFFODIL_JAVA_OPTS) else Util.start("")
   ```
     

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r390349714
 
 

 ##########
 File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Term.scala
 ##########
 @@ -746,7 +741,7 @@ trait Term
    *
    * What stops this is when the end of an enclosing element has to be next.
    */
-  final def possibleNextChildElementsInInfoset: Seq[ElementBase] = LV('possibleNextChildElementsInInfoset) {
+  def possibleNextChildElementsInInfoset: Seq[ElementBase] = LV('possibleNextChildElementsInInfoset) {
 
 Review comment:
   Any particular reason final was removed? It doesn't look like this patch changes anything to override this function.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #332: Refactor isHidden

Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #332: Refactor isHidden
URL: https://github.com/apache/incubator-daffodil/pull/332#discussion_r392967071
 
 

 ##########
 File path: daffodil-runtime1/src/main/scala/org/apache/daffodil/debugger/InteractiveDebugger.scala
 ##########
 @@ -1497,6 +1482,17 @@ class InteractiveDebugger(runner: InteractiveDebuggerRunner, eCompilers: Express
         }
       }
 
+      object InfoHidden extends DebugCommand with DebugCommandValidateZeroArgs {
+        val name = "hidden"
+        override lazy val short = "hid"
+        val desc = "display whether or not we're within a hidden state"
 
 Review comment:
   Doesn't look like this was resolved.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services