You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by "stevedlawrence (via GitHub)" <gi...@apache.org> on 2023/10/19 15:35:08 UTC
[PR] Reduce copies in VariableMap implementation [daffodil]
stevedlawrence opened a new pull request, #1097:
URL: https://github.com/apache/daffodil/pull/1097
The VariableMap currently uses a Scala Map to look up variable instances, where the key is the QName of the variable and the value is the variable instance stack.
One consideration with the data structure to be used is that it must be frequently copied each time a suspension is created. So although fast lookups are important, so are fast copies. And unfortunately, Map have relatively expensive copies. A large backing array must be created, buckets allocated, keys must be rehashed, etc. This overhead is enough that it can be a significant contribution when profiling, especially in formats with lots of suspensions.
To fix this, this replaces the Map with an Array, and each VariableRuntimeData is assigned an index into this array. Now, instead of looking up a variable by its QName, we just get its index from the VRD and access that index in the array. This has the same constant time lookup, but array copies are faster and require fewer and smaller allocations.
Similarly, during suspensions we must also create an immutable copy of each variable instance inside the array so that suspensions do not see new variables instances. Variable instances are currently represented as an ArrayBuffer, which has noticeable overhead when copying. Since this ArrayBuffer is just used as a stack, this replaces it with a Seq. This gives us similar stack-like behavior, but is immutable so copies are free. This does mean however that new Seq's must be allocated if we create a lot of newVariableInstances, but that is rare enough, and still fairly fast, that the benefit of preallocating an ArrayBuffer stack does not have significant benefits.
Note that the Seq of VariableRuntimeData is added as a member to the VariableMap since in some cases (e.g. debugging, setting external variables), we do need a way to lookup the VRD by QName. This will be relatively slow, so any performance critical sections should find the VRD during compilation and use that during runtime.
This also removes the VariableMapFactory, it doesn't add much value except a level of indirection.
In a large schema with small files and lots of suspensions, this saw around a 50% improvement in unparse performance.
DAFFODIL-2852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1367418656
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -311,31 +306,33 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
* expression
*/
def setFirstInstanceInitialValues(): Unit = {
- vTable.foreach {
- case (_, variableInstances) => {
- variableInstances(0).firstInstanceInitialValue = variableInstances(0).value
- }
+ vTable.foreach { variableInstances =>
+ variableInstances.head.firstInstanceInitialValue = variableInstances.head.value
}
}
+ /**
+ * Performance of this is linear in number of variables, this should not be used in
+ * performance critical sections.
+ */
def find(qName: GlobalQName): Option[VariableInstance] = {
- val optBuf = vTable.get(qName)
- val variab = {
- if (optBuf.isDefined)
- Some(optBuf.get.last)
- else
- None
- }
- variab
+ getVariableRuntimeData(qName).map { vrd => vTable(vrd.vmapIndex).head }
}
+ /**
+ * Performance of this is linear in number of variables, this should not be used in
+ * performance critical sections.
+ */
def qnames(): Seq[GlobalQName] = {
- vTable.toSeq.map(_._1)
+ vrds.map { _.globalQName }
Review Comment:
Yeah, I'll make it a lazy val.
I'm just thinking in general there might be cases where something is a lazy val, but it's only ever used once, so it's not worth saving and taking up memory--keeping it a def might let it be garbage collected sooner. I don't think this is one of those cases and a lazy val makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1367418073
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -431,59 +416,57 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
pstate: ParseOrUnparseState,
) = {
val varQName = vrd.globalQName
- val variableInstances = vTable.get(varQName)
- if (variableInstances.isDefined) {
- val variable = variableInstances.get.last
- variable.state match {
- case VariableSet => {
- referringContext.SDE(
- "Cannot set variable %s twice. State was: %s. Existing value: %s",
- variable.rd.globalQName,
- VariableSet,
- variable.value,
- )
- }
+ val variableInstances = vTable(vrd.vmapIndex)
+ val variable = variableInstances.head
+ variable.state match {
+ case VariableSet => {
+ referringContext.SDE(
+ "Cannot set variable %s twice. State was: %s. Existing value: %s",
+ variable.rd.globalQName,
+ VariableSet,
+ variable.value,
+ )
+ }
- case VariableRead => {
+ case VariableRead => {
+
+ /**
+ * TODO: This should be an SDE, but due to a bug (DAFFODIL-1443) in
+ * the way we evaluate escapeSchemes it could lead us to setting the
+ * variable read too early */
+ pstate.SDW(
+ WarnID.VariableSet,
+ "Cannot set variable %s after reading the default value. State was: %s. Existing value: %s",
+ variable.rd.globalQName,
+ VariableSet,
+ variable.value,
+ )
+ variable.setValue(newValue)
+ variable.setState(VariableSet)
+ }
+ case _ => {
+ vrd.direction match {
/**
- * TODO: This should be an SDE, but due to a bug (DAFFODIL-1443) in
- * the way we evaluate escapeSchemes it could lead us to setting the
- * variable read too early */
- pstate.SDW(
- WarnID.VariableSet,
- "Cannot set variable %s after reading the default value. State was: %s. Existing value: %s",
- variable.rd.globalQName,
- VariableSet,
- variable.value,
- )
- variable.setValue(newValue)
- variable.setState(VariableSet)
- }
-
- case _ => {
- vrd.direction match {
- /**
- * Due to potential race conditions regarding the setting of
- * variables via setVariable and default values in combination with
- * suspensions during unparsing, we only allow the use of either
- * setVariable statements or a default value when unparsing a
- * variable.
- */
- case VariableDirection.UnparseOnly | VariableDirection.Both
- if (vrd.maybeDefaultValueExpr.isDefined && variableInstances.get.size > 1) => {
- // Variable has an unparse direction, a default value, and a
- // newVariableInstance
- pstate.SDE(
- "Variable %s has an unparse direction and a default value, setting the variable may cause race conditions when combined with a forward referencing expression.",
- varQName,
- )
- }
- case _ => // Do nothing
+ * Due to potential race conditions regarding the setting of
Review Comment:
That PR has a lot of context for this.
Clearly not this change set's problem.
It is quite possible there are bugs in our variables implementation. We don't have enough here to suggest creating a new JIRA ticket about any of this.
I'm marking this thread "resolved".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1367068944
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -384,41 +376,34 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
case _ => // Do nothing
}
- val variableInstances = vTable.get(varQName)
- if (variableInstances.isDefined) {
- val variable = variableInstances.get.last
- variable.state match {
- case VariableRead if (variable.value.isDefined) => variable.value.getNonNullable
- case VariableDefined | VariableSet if (variable.value.isDefined) => {
- variable.setState(VariableRead)
- variable.value.getNonNullable
- }
- // This case is only hit for defineVariable's who's expression reference
- // other defineVariables with expressions. It will be hit at the start
- // of parsing, before an infoset is generated, via the
- // forceExpressionEvaluations function after which all variables should
- // have a defined value
- case VariableUndefined if (variable.rd.maybeDefaultValueExpr.isDefined) => {
- variable.setState(VariableBeingDefined)
- val res =
- DataValue.unsafeFromAnyRef(variable.rd.maybeDefaultValueExpr.get.evaluate(state))
-
- // Need to update the variable's value with the result of the
- // expression
- variable.setState(VariableRead)
- variable.setValue(res)
-
- res
- }
- case VariableBeingDefined => throw new VariableCircularDefinition(varQName, vrd)
- case VariableInProcess => throw new VariableSuspended(varQName, vrd)
- case _ => throw new VariableHasNoValue(varQName, vrd)
+ val variable = vTable(vrd.vmapIndex).head
+ variable.state match {
+ case VariableRead if (variable.value.isDefined) => variable.value.getNonNullable
+ case VariableDefined | VariableSet if (variable.value.isDefined) => {
+ variable.setState(VariableRead)
+ variable.value.getNonNullable
+ }
+ // This case is only hit for defineVariable's who's expression reference
+ // other defineVariables with expressions. It will be hit at the start
+ // of parsing, before an infoset is generated, via the
+ // forceExpressionEvaluations function after which all variables should
+ // have a defined value
+ case VariableUndefined if (variable.rd.maybeDefaultValueExpr.isDefined) => {
+ variable.setState(VariableBeingDefined)
+ val res =
+ DataValue.unsafeFromAnyRef(variable.rd.maybeDefaultValueExpr.get.evaluate(state))
+
+ // Need to update the variable's value with the result of the
+ // expression
+ variable.setState(VariableRead)
+ variable.setValue(res)
Review Comment:
> There is a codecov warning saying that daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala#L398-L399 were not covered by tests. This warning may or may not be spurious because these 2 lines are preceded by 2 lines which received no warnings and these 4 lines should have been executed together or not at all.
According to codecov it does look like these two lines are currently covered in the main branch:
https://app.codecov.io/gh/apache/daffodil/blob/main/daffodil-runtime1%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fdaffodil%2Fruntime1%2Fprocessors%2FVariableMap1.scala#L408
And I've added an Assert that confirms that these lines are in fact not being hit now.
The only thing I can think of is maybe the order of things have changed (for example, looping over a Map will give a different order than looping over an Array), and now the `evaluate()` call right before these lines throws an exception earlier than it did before and doesn't give a chance to hit these lines. I'll see if I can modify a test that get this coverage back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "tuxji (via GitHub)" <gi...@apache.org>.
tuxji commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1366939459
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/SchemaSet.scala:
##########
@@ -571,9 +571,10 @@ final class SchemaSet private (
Seq(encDFV, boDFV, binDFV, outDFV)
}
- lazy val allDefinedVariables = schemas.flatMap {
- _.defineVariables
- }
+ lazy val allDefinedVariables = schemas
+ .flatMap(_.defineVariables)
+ .union(predefinedVars)
Review Comment:
This change to `allDefinedVariables` and the change to `variableMap` in the next file below moves the lookup of `predefinedVars` from `variableMap` to `allDefinedVariables`. I do not see any change to `variableMap`'s behavior (it already retrieved `predefinedVars`) but I do see a change to `allDefinedVariables`'s behavior (it didn't retrieve `predefinedVars` before).
I grep'ed for calls of `allDefinedVariables` and found the only use in Daffodil *before this PR* was 3 lines below in `allExternalVariables`'s definition. This PR introduces 3 new uses of `allDefinedVariables` (shown below). The `indexOf`'s are no concern and the other place had already looked up predefined variables as well, so it is no concern that `allDefinedVariables` includes predefined variables as well as defined variables now.
```rg
daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/SchemaSet.scala
574: lazy val allDefinedVariables = schemas
578: lazy val allExternalVariables = allDefinedVariables.filter {
daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/SchemaSetRuntime1Mixin.scala
38: val vrds = allDefinedVariables.map { _.variableRuntimeData }
daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/VariableRuntime1Mixin.scala
49: this.schemaSet.allDefinedVariables.indexOf(this),
101: this.schemaSet.allDefinedVariables.indexOf(defv),
```
I grep'ed for calls of `allExternalVariables` and did not find any before or after this PR. If there truly are no uses of `allExternalVariables` anywhere in Daffodil, should we remove `allExternalVariables` from line 578 below?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1367209492
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -384,41 +376,34 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
case _ => // Do nothing
}
- val variableInstances = vTable.get(varQName)
- if (variableInstances.isDefined) {
- val variable = variableInstances.get.last
- variable.state match {
- case VariableRead if (variable.value.isDefined) => variable.value.getNonNullable
- case VariableDefined | VariableSet if (variable.value.isDefined) => {
- variable.setState(VariableRead)
- variable.value.getNonNullable
- }
- // This case is only hit for defineVariable's who's expression reference
- // other defineVariables with expressions. It will be hit at the start
- // of parsing, before an infoset is generated, via the
- // forceExpressionEvaluations function after which all variables should
- // have a defined value
- case VariableUndefined if (variable.rd.maybeDefaultValueExpr.isDefined) => {
- variable.setState(VariableBeingDefined)
- val res =
- DataValue.unsafeFromAnyRef(variable.rd.maybeDefaultValueExpr.get.evaluate(state))
-
- // Need to update the variable's value with the result of the
- // expression
- variable.setState(VariableRead)
- variable.setValue(res)
-
- res
- }
- case VariableBeingDefined => throw new VariableCircularDefinition(varQName, vrd)
- case VariableInProcess => throw new VariableSuspended(varQName, vrd)
- case _ => throw new VariableHasNoValue(varQName, vrd)
+ val variable = vTable(vrd.vmapIndex).head
+ variable.state match {
+ case VariableRead if (variable.value.isDefined) => variable.value.getNonNullable
+ case VariableDefined | VariableSet if (variable.value.isDefined) => {
+ variable.setState(VariableRead)
+ variable.value.getNonNullable
+ }
+ // This case is only hit for defineVariable's who's expression reference
+ // other defineVariables with expressions. It will be hit at the start
+ // of parsing, before an infoset is generated, via the
+ // forceExpressionEvaluations function after which all variables should
+ // have a defined value
+ case VariableUndefined if (variable.rd.maybeDefaultValueExpr.isDefined) => {
+ variable.setState(VariableBeingDefined)
+ val res =
+ DataValue.unsafeFromAnyRef(variable.rd.maybeDefaultValueExpr.get.evaluate(state))
+
+ // Need to update the variable's value with the result of the
+ // expression
+ variable.setState(VariableRead)
+ variable.setValue(res)
Review Comment:
> Yet another reason to always use LinkedHashMap, not regular HashMap. Deterministic behavior is your friend.
Agreed.
Turns out this was the issue. As the comment for this case in the code suggests, this code path is used to support cases where the `defaultValue` of a `defineVariable` references another variable that has a `defaultValue` that hasn't been evaluated yet. So it effectively pauses and goes to evaluate that other `defaultValue` real quick.
Due to the random nature of Maps, it was very easy for this to happen. Randomly evaluating `defaultValues` (which is essentially what we did) will very likely get into a case where one variable references some other variable whose `defaultValue` hasn't been evaluated yet.
But with this change, we now evaluate variables in the exact order that they are defined in the schema, and none of our tests happened to have a case where variables were defined in backwards order that could cause this. I've added a test that explicitly does this to make sure it works.
Note that the only other test where we fall into this block is with a circular definition of `defaultValue`s, which is detected and an exception thrown in the `evaluate()` function. So that explains why we had coverage just above these two lines where `evaluate()` is called, but not these two lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1367303241
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -431,59 +416,57 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
pstate: ParseOrUnparseState,
) = {
val varQName = vrd.globalQName
- val variableInstances = vTable.get(varQName)
- if (variableInstances.isDefined) {
- val variable = variableInstances.get.last
- variable.state match {
- case VariableSet => {
- referringContext.SDE(
- "Cannot set variable %s twice. State was: %s. Existing value: %s",
- variable.rd.globalQName,
- VariableSet,
- variable.value,
- )
- }
+ val variableInstances = vTable(vrd.vmapIndex)
+ val variable = variableInstances.head
+ variable.state match {
+ case VariableSet => {
+ referringContext.SDE(
+ "Cannot set variable %s twice. State was: %s. Existing value: %s",
+ variable.rd.globalQName,
+ VariableSet,
+ variable.value,
+ )
+ }
- case VariableRead => {
+ case VariableRead => {
+
+ /**
+ * TODO: This should be an SDE, but due to a bug (DAFFODIL-1443) in
+ * the way we evaluate escapeSchemes it could lead us to setting the
+ * variable read too early */
+ pstate.SDW(
+ WarnID.VariableSet,
+ "Cannot set variable %s after reading the default value. State was: %s. Existing value: %s",
+ variable.rd.globalQName,
+ VariableSet,
+ variable.value,
+ )
+ variable.setValue(newValue)
+ variable.setState(VariableSet)
+ }
+ case _ => {
+ vrd.direction match {
/**
- * TODO: This should be an SDE, but due to a bug (DAFFODIL-1443) in
- * the way we evaluate escapeSchemes it could lead us to setting the
- * variable read too early */
- pstate.SDW(
- WarnID.VariableSet,
- "Cannot set variable %s after reading the default value. State was: %s. Existing value: %s",
- variable.rd.globalQName,
- VariableSet,
- variable.value,
- )
- variable.setValue(newValue)
- variable.setState(VariableSet)
- }
-
- case _ => {
- vrd.direction match {
- /**
- * Due to potential race conditions regarding the setting of
- * variables via setVariable and default values in combination with
- * suspensions during unparsing, we only allow the use of either
- * setVariable statements or a default value when unparsing a
- * variable.
- */
- case VariableDirection.UnparseOnly | VariableDirection.Both
- if (vrd.maybeDefaultValueExpr.isDefined && variableInstances.get.size > 1) => {
- // Variable has an unparse direction, a default value, and a
- // newVariableInstance
- pstate.SDE(
- "Variable %s has an unparse direction and a default value, setting the variable may cause race conditions when combined with a forward referencing expression.",
- varQName,
- )
- }
- case _ => // Do nothing
+ * Due to potential race conditions regarding the setting of
Review Comment:
I didn't actually change this. This is just an indention change. I'm not sure what the comment is suggesting is the problem. Looks like Josh wrote this, @jadams-tresys, do you remember what this is about?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence merged PR #1097:
URL: https://github.com/apache/daffodil/pull/1097
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1367298136
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -311,31 +306,33 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
* expression
*/
def setFirstInstanceInitialValues(): Unit = {
- vTable.foreach {
- case (_, variableInstances) => {
- variableInstances(0).firstInstanceInitialValue = variableInstances(0).value
- }
+ vTable.foreach { variableInstances =>
+ variableInstances.head.firstInstanceInitialValue = variableInstances.head.value
}
}
+ /**
+ * Performance of this is linear in number of variables, this should not be used in
+ * performance critical sections.
+ */
def find(qName: GlobalQName): Option[VariableInstance] = {
- val optBuf = vTable.get(qName)
- val variab = {
- if (optBuf.isDefined)
- Some(optBuf.get.last)
- else
- None
- }
- variab
+ getVariableRuntimeData(qName).map { vrd => vTable(vrd.vmapIndex).head }
}
+ /**
+ * Performance of this is linear in number of variables, this should not be used in
+ * performance critical sections.
+ */
def qnames(): Seq[GlobalQName] = {
- vTable.toSeq.map(_._1)
+ vrds.map { _.globalQName }
Review Comment:
Yep, will change. Though on second though, maybe one thing to consider is that making things a lazy val will keep the memory around. Memory seems to be our bottle neck, so maybe for things expected to be accessed rarely or only once, it's better to just recalculate it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1367284096
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -257,26 +269,18 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
* VariableInstances are mutable and cannot safely be shared across threads
*/
def copy(): VariableMap = {
- val table = vTable.map {
- case (k: GlobalQName, variableInstances: ArrayBuffer[VariableInstance]) => {
- val newBuf = variableInstances.map { _.copy() }
- (k, newBuf)
- }
+ val table = vTable.map { variableInstances =>
+ val newBuf = variableInstances.map { _.copy() }
+ newBuf
}
- new VariableMap(table)
+ new VariableMap(vrds, table)
}
- def topLevelInstances(): VariableMap = {
- val table = vTable.map {
- case (k: GlobalQName, variableInstances: ArrayBuffer[VariableInstance]) => {
- val newBuf = new ArrayBuffer[VariableInstance]()
- newBuf.append(variableInstances.last)
- (k, newBuf)
- }
- }
-
- new VariableMap(table)
+ def cloneForSuspension(): VariableMap = {
+ val newTable = new Array[Seq[VariableInstance]](vTable.size)
+ Array.copy(vTable, 0, newTable, 0, vTable.size)
+ new VariableMap(vrds, newTable)
Review Comment:
Yeah, I imagine there's room for improvement. The key issue is that suspensions currently need a pointer to the current instance of each variable, so that if later newVariableInstances happen, then those suspensions don't see that. And we need it for all variables since we don't know which variables a suspension will need.
We technically don't need the entire stack like this does, we really just need the top of the stack. So we could have done something like this:
```scala
val newTable = vTable.map { instances => Seq(instances.head) }
new VariableMap(vrds, newTable)
```
But that allocates another Seq, and avoiding allocations was the main purpose of this. And as long as suspensions aren't pushing/popping variable instances (which I don't think they should be able to do), this should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "jadams-tresys (via GitHub)" <gi...@apache.org>.
jadams-tresys commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1367342148
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -431,59 +416,57 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
pstate: ParseOrUnparseState,
) = {
val varQName = vrd.globalQName
- val variableInstances = vTable.get(varQName)
- if (variableInstances.isDefined) {
- val variable = variableInstances.get.last
- variable.state match {
- case VariableSet => {
- referringContext.SDE(
- "Cannot set variable %s twice. State was: %s. Existing value: %s",
- variable.rd.globalQName,
- VariableSet,
- variable.value,
- )
- }
+ val variableInstances = vTable(vrd.vmapIndex)
+ val variable = variableInstances.head
+ variable.state match {
+ case VariableSet => {
+ referringContext.SDE(
+ "Cannot set variable %s twice. State was: %s. Existing value: %s",
+ variable.rd.globalQName,
+ VariableSet,
+ variable.value,
+ )
+ }
- case VariableRead => {
+ case VariableRead => {
+
+ /**
+ * TODO: This should be an SDE, but due to a bug (DAFFODIL-1443) in
+ * the way we evaluate escapeSchemes it could lead us to setting the
+ * variable read too early */
+ pstate.SDW(
+ WarnID.VariableSet,
+ "Cannot set variable %s after reading the default value. State was: %s. Existing value: %s",
+ variable.rd.globalQName,
+ VariableSet,
+ variable.value,
+ )
+ variable.setValue(newValue)
+ variable.setState(VariableSet)
+ }
+ case _ => {
+ vrd.direction match {
/**
- * TODO: This should be an SDE, but due to a bug (DAFFODIL-1443) in
- * the way we evaluate escapeSchemes it could lead us to setting the
- * variable read too early */
- pstate.SDW(
- WarnID.VariableSet,
- "Cannot set variable %s after reading the default value. State was: %s. Existing value: %s",
- variable.rd.globalQName,
- VariableSet,
- variable.value,
- )
- variable.setValue(newValue)
- variable.setState(VariableSet)
- }
-
- case _ => {
- vrd.direction match {
- /**
- * Due to potential race conditions regarding the setting of
- * variables via setVariable and default values in combination with
- * suspensions during unparsing, we only allow the use of either
- * setVariable statements or a default value when unparsing a
- * variable.
- */
- case VariableDirection.UnparseOnly | VariableDirection.Both
- if (vrd.maybeDefaultValueExpr.isDefined && variableInstances.get.size > 1) => {
- // Variable has an unparse direction, a default value, and a
- // newVariableInstance
- pstate.SDE(
- "Variable %s has an unparse direction and a default value, setting the variable may cause race conditions when combined with a forward referencing expression.",
- varQName,
- )
- }
- case _ => // Do nothing
+ * Due to potential race conditions regarding the setting of
Review Comment:
I remember working on this for a while and it needs a lengthy test case in order to hit this error message:
https://github.com/apache/daffodil/blob/716e5336621d6ec645eb4a3d3b47b12680a914ff/daffodil-test/src/test/resources/org/apache/daffodil/section07/variables/variables.tdml#L260
I cannot remember if we get indeterminate behavior from this test case if we remove the error check here and I'm struggling to even understand how my test is set up as it has several layers of nesting. Unfortunately I don't have time to dig into this, but there might be some useful comments in the pull request where this got merged:
https://github.com/apache/daffodil/pull/473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1367226350
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -257,26 +269,18 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
* VariableInstances are mutable and cannot safely be shared across threads
*/
def copy(): VariableMap = {
- val table = vTable.map {
- case (k: GlobalQName, variableInstances: ArrayBuffer[VariableInstance]) => {
- val newBuf = variableInstances.map { _.copy() }
- (k, newBuf)
- }
+ val table = vTable.map { variableInstances =>
+ val newBuf = variableInstances.map { _.copy() }
+ newBuf
}
- new VariableMap(table)
+ new VariableMap(vrds, table)
}
- def topLevelInstances(): VariableMap = {
- val table = vTable.map {
- case (k: GlobalQName, variableInstances: ArrayBuffer[VariableInstance]) => {
- val newBuf = new ArrayBuffer[VariableInstance]()
- newBuf.append(variableInstances.last)
- (k, newBuf)
- }
- }
-
- new VariableMap(table)
+ def cloneForSuspension(): VariableMap = {
+ val newTable = new Array[Seq[VariableInstance]](vTable.size)
+ Array.copy(vTable, 0, newTable, 0, vTable.size)
+ new VariableMap(vrds, newTable)
Review Comment:
This is kind of interesting.
Cloning for suspension doesn't need to copy the whole stack for each variable as only the top of stack is relevant to evaluating an expression to obtain its value. (.... except then if unparsing resumes from there maybe...?)
Exploring this is beyond the scope of this change set, but we do need to think about whether "clone the state" is really needed.
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -311,31 +306,33 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
* expression
*/
def setFirstInstanceInitialValues(): Unit = {
- vTable.foreach {
- case (_, variableInstances) => {
- variableInstances(0).firstInstanceInitialValue = variableInstances(0).value
- }
+ vTable.foreach { variableInstances =>
+ variableInstances.head.firstInstanceInitialValue = variableInstances.head.value
}
}
+ /**
+ * Performance of this is linear in number of variables, this should not be used in
+ * performance critical sections.
+ */
def find(qName: GlobalQName): Option[VariableInstance] = {
- val optBuf = vTable.get(qName)
- val variab = {
- if (optBuf.isDefined)
- Some(optBuf.get.last)
- else
- None
- }
- variab
+ getVariableRuntimeData(qName).map { vrd => vTable(vrd.vmapIndex).head }
Review Comment:
Isn't a hash map of qname -> vmapIndex just a constant table that can be computed once we have all the variable declarations? So this can be constant time. Your representation avoids having to call this much as a VRD knows its index, so getting from VRD to variable on top of stack is constant time, but I think this find can also be constant time.
This would only be important if there were more than a trivial number of variables. Linear search may in fact be the fastest thing. I think most cases the number of variables is < 6.
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -431,59 +416,57 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
pstate: ParseOrUnparseState,
) = {
val varQName = vrd.globalQName
- val variableInstances = vTable.get(varQName)
- if (variableInstances.isDefined) {
- val variable = variableInstances.get.last
- variable.state match {
- case VariableSet => {
- referringContext.SDE(
- "Cannot set variable %s twice. State was: %s. Existing value: %s",
- variable.rd.globalQName,
- VariableSet,
- variable.value,
- )
- }
+ val variableInstances = vTable(vrd.vmapIndex)
+ val variable = variableInstances.head
+ variable.state match {
+ case VariableSet => {
+ referringContext.SDE(
+ "Cannot set variable %s twice. State was: %s. Existing value: %s",
+ variable.rd.globalQName,
+ VariableSet,
+ variable.value,
+ )
+ }
- case VariableRead => {
+ case VariableRead => {
+
+ /**
+ * TODO: This should be an SDE, but due to a bug (DAFFODIL-1443) in
+ * the way we evaluate escapeSchemes it could lead us to setting the
+ * variable read too early */
+ pstate.SDW(
+ WarnID.VariableSet,
+ "Cannot set variable %s after reading the default value. State was: %s. Existing value: %s",
+ variable.rd.globalQName,
+ VariableSet,
+ variable.value,
+ )
+ variable.setValue(newValue)
+ variable.setState(VariableSet)
+ }
+ case _ => {
+ vrd.direction match {
/**
- * TODO: This should be an SDE, but due to a bug (DAFFODIL-1443) in
- * the way we evaluate escapeSchemes it could lead us to setting the
- * variable read too early */
- pstate.SDW(
- WarnID.VariableSet,
- "Cannot set variable %s after reading the default value. State was: %s. Existing value: %s",
- variable.rd.globalQName,
- VariableSet,
- variable.value,
- )
- variable.setValue(newValue)
- variable.setState(VariableSet)
- }
-
- case _ => {
- vrd.direction match {
- /**
- * Due to potential race conditions regarding the setting of
- * variables via setVariable and default values in combination with
- * suspensions during unparsing, we only allow the use of either
- * setVariable statements or a default value when unparsing a
- * variable.
- */
- case VariableDirection.UnparseOnly | VariableDirection.Both
- if (vrd.maybeDefaultValueExpr.isDefined && variableInstances.get.size > 1) => {
- // Variable has an unparse direction, a default value, and a
- // newVariableInstance
- pstate.SDE(
- "Variable %s has an unparse direction and a default value, setting the variable may cause race conditions when combined with a forward referencing expression.",
- varQName,
- )
- }
- case _ => // Do nothing
+ * Due to potential race conditions regarding the setting of
Review Comment:
This needs a bit more explanation. What exactly are you saying can cause a race here? It's all state for one thread. So how can there be a race? There has to be something non-deterministic going on.
If two suspensions depend on one variable, one is evaluated and gets the value before the set variable actually executes... so the default value has been consumed before the setVariable runs. When you setVariable it will fail. If these can happen in the opposite order, then yes that's a race, and that's due to non-deterministic behavior somewhere in the suspension scheduling.
I realize suspensions get evaluated in a humanly non-predictable order, but it should be deterministic.
If not we should make it deterministic so that any given scenario is repeatable, and may be incorrect per the setVariable happening after the read of the default has already occurred, but this should happen every time or never, not sporadically.
This change set is not about this problem however, so we can park this as a separate bug.
Do we check for this and at least warn about it at compile time? To me if you use newVariableInstance, the variable declaration should not have a default value, I think.
##########
daffodil-core/src/test/scala/org/apache/daffodil/core/util/TestUtils.scala:
##########
@@ -389,7 +388,7 @@ class Fakes private () {
override def isError: Boolean = false
override def tunables: DaffodilTunables = DaffodilTunables()
- override def variableMap: VariableMap = VariableMapFactory.create(Nil)
+ override def variableMap: VariableMap = VariableMap(Nil)
Review Comment:
Ah. I was confusing this with VariableBox, which is associated with unparser suspensions in some way.
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -311,31 +306,33 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
* expression
*/
def setFirstInstanceInitialValues(): Unit = {
- vTable.foreach {
- case (_, variableInstances) => {
- variableInstances(0).firstInstanceInitialValue = variableInstances(0).value
- }
+ vTable.foreach { variableInstances =>
+ variableInstances.head.firstInstanceInitialValue = variableInstances.head.value
}
}
+ /**
+ * Performance of this is linear in number of variables, this should not be used in
+ * performance critical sections.
+ */
def find(qName: GlobalQName): Option[VariableInstance] = {
- val optBuf = vTable.get(qName)
- val variab = {
- if (optBuf.isDefined)
- Some(optBuf.get.last)
- else
- None
- }
- variab
+ getVariableRuntimeData(qName).map { vrd => vTable(vrd.vmapIndex).head }
}
+ /**
+ * Performance of this is linear in number of variables, this should not be used in
+ * performance critical sections.
+ */
def qnames(): Seq[GlobalQName] = {
- vTable.toSeq.map(_._1)
+ vrds.map { _.globalQName }
Review Comment:
Isn't this a constant, as in could be lazy val?
##########
daffodil-core/src/test/scala/org/apache/daffodil/core/util/TestUtils.scala:
##########
@@ -389,7 +388,7 @@ class Fakes private () {
override def isError: Boolean = false
override def tunables: DaffodilTunables = DaffodilTunables()
- override def variableMap: VariableMap = VariableMapFactory.create(Nil)
+ override def variableMap: VariableMap = VariableMap(Nil)
Review Comment:
There was some reason for the factory..... I can't recall. Maybe serialization and hashmap related? In any case if tests pass now, and you can serialize/deserialize processors it's no longer needed.
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -233,21 +240,26 @@ final class VariableBox(initialVMap: VariableMap) {
* The DPath implementation must be made to implement the
* no-set-after-default-value-has-been-read behavior. This requires that reading the variables causes a state transition.
*
- * The implementation of the VariabeMap uses ArrayBuffers essentially as a
- * stack, as they allow for easy serialization unlike the custom MStack classes
- * we use elsewhere. Scala's mutable Stack is deprecated in 2.12
+ * The VariableMap implementation essentially uses Seq as a stack, as they allow for easy
+ * serialization unlike the custom MStack classes we use elsewhere. Note that Scala's mutable
Review Comment:
Does this suggest a revisit (not in this change set) of MStack generally?
The point of MStack was to avoid allocating objects when pushing value classes to/from stacks. If that's not worth it because we encounter overhead duplicating the MStacks.... Maybe they should be reimplemented in a simpler way or just go away entirely?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1367272374
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -233,21 +240,26 @@ final class VariableBox(initialVMap: VariableMap) {
* The DPath implementation must be made to implement the
* no-set-after-default-value-has-been-read behavior. This requires that reading the variables causes a state transition.
*
- * The implementation of the VariabeMap uses ArrayBuffers essentially as a
- * stack, as they allow for easy serialization unlike the custom MStack classes
- * we use elsewhere. Scala's mutable Stack is deprecated in 2.12
+ * The VariableMap implementation essentially uses Seq as a stack, as they allow for easy
+ * serialization unlike the custom MStack classes we use elsewhere. Note that Scala's mutable
Review Comment:
Yeah, it's definitely possible we should take a look at MStack. I didn't see it pop up in my profiling, but I didn't do a lot and just focuses and the really big stuff like this variable map. I hope to do some more extensive profiling in the next release so I'll make sure to keep any eye out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1367176522
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -384,41 +376,34 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
case _ => // Do nothing
}
- val variableInstances = vTable.get(varQName)
- if (variableInstances.isDefined) {
- val variable = variableInstances.get.last
- variable.state match {
- case VariableRead if (variable.value.isDefined) => variable.value.getNonNullable
- case VariableDefined | VariableSet if (variable.value.isDefined) => {
- variable.setState(VariableRead)
- variable.value.getNonNullable
- }
- // This case is only hit for defineVariable's who's expression reference
- // other defineVariables with expressions. It will be hit at the start
- // of parsing, before an infoset is generated, via the
- // forceExpressionEvaluations function after which all variables should
- // have a defined value
- case VariableUndefined if (variable.rd.maybeDefaultValueExpr.isDefined) => {
- variable.setState(VariableBeingDefined)
- val res =
- DataValue.unsafeFromAnyRef(variable.rd.maybeDefaultValueExpr.get.evaluate(state))
-
- // Need to update the variable's value with the result of the
- // expression
- variable.setState(VariableRead)
- variable.setValue(res)
-
- res
- }
- case VariableBeingDefined => throw new VariableCircularDefinition(varQName, vrd)
- case VariableInProcess => throw new VariableSuspended(varQName, vrd)
- case _ => throw new VariableHasNoValue(varQName, vrd)
+ val variable = vTable(vrd.vmapIndex).head
+ variable.state match {
+ case VariableRead if (variable.value.isDefined) => variable.value.getNonNullable
+ case VariableDefined | VariableSet if (variable.value.isDefined) => {
+ variable.setState(VariableRead)
+ variable.value.getNonNullable
+ }
+ // This case is only hit for defineVariable's who's expression reference
+ // other defineVariables with expressions. It will be hit at the start
+ // of parsing, before an infoset is generated, via the
+ // forceExpressionEvaluations function after which all variables should
+ // have a defined value
+ case VariableUndefined if (variable.rd.maybeDefaultValueExpr.isDefined) => {
+ variable.setState(VariableBeingDefined)
+ val res =
+ DataValue.unsafeFromAnyRef(variable.rd.maybeDefaultValueExpr.get.evaluate(state))
+
+ // Need to update the variable's value with the result of the
+ // expression
+ variable.setState(VariableRead)
+ variable.setValue(res)
Review Comment:
re: _looping over a Map will give a different order than looping over an Array_
Yet another reason to **always** use LinkedHashMap, not regular HashMap. Deterministic behavior is your friend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1367294476
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -311,31 +306,33 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
* expression
*/
def setFirstInstanceInitialValues(): Unit = {
- vTable.foreach {
- case (_, variableInstances) => {
- variableInstances(0).firstInstanceInitialValue = variableInstances(0).value
- }
+ vTable.foreach { variableInstances =>
+ variableInstances.head.firstInstanceInitialValue = variableInstances.head.value
}
}
+ /**
+ * Performance of this is linear in number of variables, this should not be used in
+ * performance critical sections.
+ */
def find(qName: GlobalQName): Option[VariableInstance] = {
- val optBuf = vTable.get(qName)
- val variab = {
- if (optBuf.isDefined)
- Some(optBuf.get.last)
- else
- None
- }
- variab
+ getVariableRuntimeData(qName).map { vrd => vTable(vrd.vmapIndex).head }
Review Comment:
Yeah, I considered doing that, but the only places we do a qname lookup are in layer compilation (if they use variables), CLI debugging, or setting external variables. Only that last one *really* matters for performance, but 1) setting external variables is fairly rare and 2) setExternalVariables has more compliated logic than a hash lookup--it considers local name, detects ambiguities if there's no namspace, etc. As you point out linear search is probably fine for these cases with the usual few number of variables.
In reality, these functions probably aren't that slow, but I added the comment because I don't want us to accidentally use them in actual parsing/unparsing logic or something where speed really is critical.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1367406274
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/VariableMap1.scala:
##########
@@ -311,31 +306,33 @@ class VariableMap private (vTable: Map[GlobalQName, ArrayBuffer[VariableInstance
* expression
*/
def setFirstInstanceInitialValues(): Unit = {
- vTable.foreach {
- case (_, variableInstances) => {
- variableInstances(0).firstInstanceInitialValue = variableInstances(0).value
- }
+ vTable.foreach { variableInstances =>
+ variableInstances.head.firstInstanceInitialValue = variableInstances.head.value
}
}
+ /**
+ * Performance of this is linear in number of variables, this should not be used in
+ * performance critical sections.
+ */
def find(qName: GlobalQName): Option[VariableInstance] = {
- val optBuf = vTable.get(qName)
- val variab = {
- if (optBuf.isDefined)
- Some(optBuf.get.last)
- else
- None
- }
- variab
+ getVariableRuntimeData(qName).map { vrd => vTable(vrd.vmapIndex).head }
}
+ /**
+ * Performance of this is linear in number of variables, this should not be used in
+ * performance critical sections.
+ */
def qnames(): Seq[GlobalQName] = {
- vTable.toSeq.map(_._1)
+ vrds.map { _.globalQName }
Review Comment:
This is just in the variable map though? Isn't that a constant data structure?
Either way no biggie. It's documented as linear so all good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Reduce copies in VariableMap implementation [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1097:
URL: https://github.com/apache/daffodil/pull/1097#discussion_r1367049322
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/dsom/SchemaSet.scala:
##########
@@ -571,9 +571,10 @@ final class SchemaSet private (
Seq(encDFV, boDFV, binDFV, outDFV)
}
- lazy val allDefinedVariables = schemas.flatMap {
- _.defineVariables
- }
+ lazy val allDefinedVariables = schemas
+ .flatMap(_.defineVariables)
+ .union(predefinedVars)
Review Comment:
Yeah, nothing used `allDefinedVariables`, and I figured it made sense to start using it, and I think it's more useful it if includes the predefined vars.
The only use case I can think of for `allExternalVariables` is if something like a VS Code extension wants to know which variables could be set externally. I don't think there's any other way to easily get that information now. I thought we had something like that in the `VariableMap` class but we only have functions get all `QName`s, or get a single `VariableRuntimeData` given a `QName`. But those functions could be combined like this:
```scala
val externalQNames = vmap.qnames.filter { qn => vmap.getVariableRuntimeData(qn).get.external }
```
It's not super efficient, but performance should be fine enough for an IDE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org