You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by "shanedell (via GitHub)" <gi...@apache.org> on 2025/03/19 15:31:29 UTC
[PR] Add left over data message to debugger and extension [daffodil-vscode]
shanedell opened a new pull request, #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204
Add left over data message to debugger and extension
- After a parse completes, the debugger will check if there is data left over.
- When data is left over the debugger will log an error message to the console.
- The debugger also sends data, bitPos1b, bytePos1b, leftOverBits and message to the extension.
- Currently the extension parses the sent over data and shows an error message with the message the debugger sent.
Closes #974
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2752136874
All Dump.dump does is convert non-printable characters into special unicode characters that are visible. It seems like something can't handle those special unicode characters.
For example, if you skip Dump.dump and you do this to create the dump string:
```scala
val bytes = Array[Byte](0xE2.toByte, 0x90.toByte, 0x80.toByte)
val dumpString = new String(bytes, StandardCharsets.UTF_8)
```
That creates a string with just the special `␀` character.
I tried tracking it down, but it's not clear what's breaking with these unicode symbols. It could be a lot of places. Maybe the DAP sender doesn't support it, maybe the DAP receiver doesn't, maybe it can't extract it from the DAP protocol, maybe VSCode can't display these unicode symbols. Or maybe there's en encoding mismatch between sender and receiver.
I don't have VS Code build setup so it's hard to test where things are failing, but my guess is something can't handle these more complex unicode characters.
Note that special unicode characters are nothing special, and can appear in things like infosets, so this isn't necessarily specific to left over data and debug messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#discussion_r2006032673
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Utils.scala:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.debugger.dap
+
+import java.nio.charset.StandardCharsets
+
+/** Data Left Over Utils */
+object DataLeftOverUtils {
Review Comment:
I could have sworn Java had a built-in way to read part of a file, but I can't find what I was thikning of. Though, looks like you could do something like
```scala
// calculate number of bytes to dump beyond the end of the parse, maybe
// something like min(10, bytesRemaining)
val numBytes = ...
// read to byte array starting where the parse ended
val f = new RandomAccessFile(...)
val bytes = new Array[Byte](dumpSize)
f.seek(bytePos1b - 1)
f.readFully(bytes)
// convert those bytes to hex/binary
val hexLines = DataDumper.dump(HexKind, loc.bitPos1b - 1, numBytes * 8, ByteBuffer.wrap(bytes), includeHeadingLine = false)
val textLines = DataDumper.dump(TextKind, loc.bitPos1b - 1, numBytes * 8, ByteBuffer.wrap(bytes), includeHeadingLine = false)
```
That's not tested, but I was thinking something along those lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2755391368
@stevedlawrence I pushed that update up, so everything should be good now if you wanted to re-review. @rthomas320 You should now be able to pull down the updated code and test it, you should get the logging message and popup for left over data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#discussion_r2007914230
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Utils.scala:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.debugger.dap
+
+import java.nio.charset.StandardCharsets
+
+/** Data Left Over Utils */
+object DataLeftOverUtils {
Review Comment:
That is the expected output of the dump function--it converts non-visible characters to visible glyphs. For example, space is converted to `␣`, NUL characters are conerted to `␀`. This makes it easier for users to visualize what the non-printable characters are.
Maybe the logging system can't handle those more complex characters or they need to be escaped somehow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "hdalsania (via GitHub)" <gi...@apache.org>.
hdalsania commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2748613104
> @JeremyYao The error doesn't show up in the Debug Console, it will be Terminal output. I am wondering if its not showing in there and that's why the popup is not coming up. Currently I am facing that same issue with some of the changes I have made
Errors are showing up in Terminal Output but popup is only showing up for data left over error. Other two test cases from Jeremy's post do not show the popup window but error shows up in the Terminal Output window.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on code in PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#discussion_r2005981537
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Utils.scala:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.debugger.dap
+
+import java.nio.charset.StandardCharsets
+
+/** Data Left Over Utils */
+object DataLeftOverUtils {
Review Comment:
So I am not sure how to use the DataDumper alongside Files.read, would you have any recommendations there on what to look at doing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on code in PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#discussion_r2007917494
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Utils.scala:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.debugger.dap
+
+import java.nio.charset.StandardCharsets
+
+/** Data Left Over Utils */
+object DataLeftOverUtils {
Review Comment:
That is what I am assuming since the message is correct and when using `println` it can display the message perfectly fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#discussion_r2007791188
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Utils.scala:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.debugger.dap
+
+import java.nio.charset.StandardCharsets
+
+/** Data Left Over Utils */
+object DataLeftOverUtils {
Review Comment:
Nothing jumps about at me, but I'm not too familiar with the details of how the vscode extensions and dap stuff works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2754900047
> I pulled in the code for PR #1204 Left over data message. I am not seeing the left over data message on Windows 10. Also the Omega Edit server window is no longer closing after debugging finishes.
@rthomas320 Currently @stevedlawrence and I are debugging the issue with the left over message not showing when using DataDumper so that should hopefully get resolved soon. However, the issue with `omega-edit` is not related to this PR, that might be another issue that needs to created, it might have broken with @scholarsmate's latest merged PR possibly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2754579469
I've managed to get the extension installed and I can reproduce the issue. I can confirm that it does look like it's somehow related to `Dump.dump`--if I manually create strings that output ` ␀` it works fine, so Dump.dump is does seem to be doing something weird. I'll continue investigating and let you know what I find.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell merged PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "stricklandrbls (via GitHub)" <gi...@apache.org>.
stricklandrbls commented on code in PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#discussion_r2005997323
##########
src/daffodilDebugger/daffodil.ts:
##########
@@ -83,13 +96,15 @@ export class DaffodilDebugEvent<
> implements DebugSessionCustomEvent
{
readonly session: DebugSession
+
constructor(
readonly event: E,
readonly body: B
) {
this.session = debug.activeDebugSession!
}
- asEditorMessage(): { command: string; data: any } {
+
+ asObject(): { command: string; data: any } {
Review Comment:
Looks like the `{command: string; data: any}` could have a type definition since this signature is being called in multiple places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2752318257
The core issue has nothing to do with the DataDumper except that is is more likely to output weird unicode characters. But left over data can contain any bytes and so could just as easily contain bytes the break the logger or dapEvent thing.
Can you confirm if it's the Logger or dapEvents.send that is breaking? If you remove the logger and do this does it still fail?
```scala
dapEvents.send(DataLeftOverEvent(loc.bitPos1b, loc.bytePos1b, leftOverBits, message)
```
Or if you just do:
```scala
Logger[IO].error(message)
```
Does it correctly log the message?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2752274035
@stevedlawrence So its failing I believe right at:
```scala
Logger[IO].error(message) *> dapEvents.send(
DataLeftOverEvent(loc.bitPos1b, loc.bytePos1b, leftOverBits, message)
) *> IO.unit
```
It seems that neither are working, meaning its both not being logged or forwarded to the extension. I was able to verify this when setting a breakpoint in the extension code where it would catch the event being sent over, this breakpoint would never get hit. I tried to clean out special characters after getting the message but that didn't seem to help anything. At this point I am unsure of what I should do to try to resolve the issue other than not using `DataDumper` but not sure if that's something we want
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2751709615
@stevedlawrence Can you re-review this? I pushed up my changes that use `DataDumper`, however the message is not currently showing in the log or the pop up window. This seems to be an issue caused by `DataDumper` because if I remove the `DataDumper` related parts and only add the dataHex then the message and popup works fine. Is there an error I could be encountering with `DataDumper` that you are aware of that might not be easy to spot? I have been working on this for a couple days and can't seem to figure out what is going on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2751785241
> Can you re-review this?
Sure thing, I'll take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on code in PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#discussion_r2007784261
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Utils.scala:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.debugger.dap
+
+import java.nio.charset.StandardCharsets
+
+/** Data Left Over Utils */
+object DataLeftOverUtils {
Review Comment:
@stevedlawrence do you notice anything wrong with the way I am logging that is causing it not to happen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#discussion_r2007782008
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Utils.scala:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.debugger.dap
+
+import java.nio.charset.StandardCharsets
+
+/** Data Left Over Utils */
+object DataLeftOverUtils {
Review Comment:
Yep, looks good to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on code in PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#discussion_r2010732074
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Utils.scala:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.debugger.dap
+
+import java.nio.charset.StandardCharsets
+
+/** Data Left Over Utils */
+object DataLeftOverUtils {
Review Comment:
@stevedlawrence So I think I have something good locally, but it doesn't not work fully if I include the DataDumper stuff specifically. Do you know what could be causing my issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2752329042
@stevedlawrence It is both. If I remove the logger and do this:
```scala
dapEvents.send(DataLeftOverEvent(loc.bitPos1b, loc.bytePos1b, leftOverBits, message))
```
The event is not forwarded, thus not caught by the extension so the breakpoint is still missed.
If I just do:
```scala
Logger[IO].error(message)
```
Nothing is logged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#discussion_r2003807709
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Parse.scala:
##########
@@ -754,12 +771,14 @@ object Parse {
Parse(
args.schemaPath,
in,
+ FileUtils.readFileToByteArray(args.dataPath.toFile),
Review Comment:
This will prevent you from being able to support files larger than 2GB. If this is important, you might want to just pass in the `File` and then the functions can call things like `Files.size()`, or `Files.read()` with offset to read just the parts of the file you want to output in diagnostics. That would avoid the 2GB limitation.
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Utils.scala:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.debugger.dap
+
+import java.nio.charset.StandardCharsets
+
+/** Data Left Over Utils */
+object DataLeftOverUtils {
Review Comment:
This seems a bit text-centric, which things like skipping whitespace. If there's left over data, spaces can be very important to understanding what is left over and usually want to to be output. And if data is binary, then users just want to see raw bytes. You might want to look at what Daffodil does to output left over data for diagnostics for inspiration.
https://github.com/apache/daffodil/blob/main/daffodil-cli/src/main/scala/org/apache/daffodil/cli/Main.scala#L1339
Daffodil has a special "Dump" class we use in the CLI, it's not technically part of the public API, can you are still free to use it, it hasn't changed in a very long time to it should be pretty stable.
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Parse.scala:
##########
@@ -104,16 +106,31 @@ object Parse {
case "json" => new JsonInfosetOutputter(os, true)
}
+ val dataInputStream = new InputSourceDataInputStream(data)
+ val parseResult = dp.parse(dataInputStream, infosetOutputter)
+ val loc = parseResult.location()
+
val parse =
IO.interruptibleMany(
- dp.parse(
- new InputSourceDataInputStream(data),
- infosetOutputter
- )
+ parseResult
// WARNING: parse doesn't close the OutputStream, so closed below
).ensureOr(res => new Parse.Exception(res.getDiagnostics.toList))(res => !res.isError)
.guarantee(IO(os.close) *> done.set(true))
.void
+ .flatMap { _ =>
+ val bitsRead = loc.bitPos1b - 1
+ val leftOverBits = (dataContents.length - (loc.bytePos1b - 1)) * 8
+
+ if (bitsRead > 0 && leftOverBits > 0) {
Review Comment:
Why the `bitsRead > 0` condition? It's legal for a DFDL schema to successfully parse but not consume any data, so that bitsRead will be zero. If were was data to consume it this case, it feels like we would still want the left over data message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on code in PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#discussion_r2007771766
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Utils.scala:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.debugger.dap
+
+import java.nio.charset.StandardCharsets
+
+/** Data Left Over Utils */
+object DataLeftOverUtils {
Review Comment:
@stevedlawrence So inside of Parse.scala, under the `flatMap` section, I currently have this:
```scala
val leftOverBits = (dataSize - (loc.bytePos1b - 1)) * 8
if (leftOverBits > 0) {
val message = DataLeftOverUtils.getMessage(dataFilePath, loc.bitPos1b, loc.bytePos1b, leftOverBits)
Logger[IO].error(message) *> dapEvents.send(
DataLeftOverEvent(loc.bitPos1b, loc.bytePos1b, leftOverBits, message)
) *> IO.unit
} else {
IO.unit
}
```
then in Utils.scala I now have:
```scala
package org.apache.daffodil.debugger.dap
import java.io.RandomAccessFile
import java.nio.ByteBuffer
import java.nio.file.Path
import org.apache.daffodil.io.DataDumper
/** Data Left Over Utils */
object DataLeftOverUtils {
/** Get message based on dataFilePath, bitPos1bm bytePos1b and leftOverBits */
def getMessage(dataFilePath: Path, bitPos1b: Long, bytePos1b: Long, leftOverBits: Long): String = {
val Dump = new DataDumper
val numBytes = math.min(8, (leftOverBits / 8)).toInt
// read to byte array starting where the parse ended
val f = new RandomAccessFile(dataFilePath.toString(), "r")
val bytes = new Array[Byte](numBytes)
f.seek(bytePos1b - 1)
f.readFully(bytes)
f.close()
// convert those bytes to hex/binary
val dumpString = Dump
.dump(
Dump.TextOnly(Some("utf-8")),
bitPos1b - 1,
numBytes * 8,
ByteBuffer.wrap(bytes),
includeHeadingLine = false
)
.mkString("\n")
val dataText = s"\nLeft over data (UTF-8) starting at byte ${bytePos1b} is: (${dumpString}...)"
val dataHex = s"\nLeft over data (Hex) starting at byte ${bytePos1b} is: (0x${bytes.map { a =>
f"$a%02x"
}.mkString}...)"
val leftOverDataMessage =
s"Left over data. Consumed ${bitPos1b - 1} bit(s) with ${leftOverBits} bit(s) remaining." + dataHex + dataText
leftOverDataMessage
}
}
```
I seem to get the correct message, because if I add a `println(message)` I get the message I want, however for some reason its not getting logged which in turn means it's not getting sent to the extension. Does this look okay to you though how I worked around things? I am still trying to figure out the not logging kink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2743623089
@JeremyYao The error doesn't show up in the Debug Console, it will be Terminal output. I am wondering if its not showing in there and that's why the popup is not coming up. Currently I am facing that same issue with some of the changes I have made
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#discussion_r2012428860
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Utils.scala:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.debugger.dap
+
+import java.nio.charset.StandardCharsets
+
+/** Data Left Over Utils */
+object DataLeftOverUtils {
Review Comment:
> But I figure something else is wrong because that isn't the same thing daffodil-cli shows, daffodil-cli doesn't have the ␀␀
I've done some testing and I can get the Daffodil CLI to output the special `␀` characters. I'm guessing the difference is just how many bytes you dump. The Daffodil CLI only dumps 8 bytes at most, which is the same size as the string `small to`, so the NUL bytes will not be dumped by the CLI. If the NUL bytes are closer to where the parse ended, you should see them in the CLI output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2752006302
> > Can you re-review this?
>
> Sure thing, I'll take a look.
@stevedlawrence I just tested and if I remove these lines
```scala
val Dump = new DataDumper
val dumpString =
if (bytes.length > 0)
Dump
.dump(
Dump.TextOnly(Some("utf-8")),
0,
bytes.length * 8,
ByteBuffer.wrap(bytes),
includeHeadingLine = false
)
.mkString("\n")
else ""
```
then do
```scala
val dumpString = new String(bytes, StandardCharsets.UTF_8)
```
I get this output:
```bash
2025-03-25 13:21:10,199 [io-compute-7] ERROR o.a.d.d.d.Parse - Left over data. Consumed 840 bit(s), with at least 405840 bit(s) remaining.
Left over data (Hex) starting at byte 106 is: (0x736d616c6c20746f...)
Left over data (UTF-8) starting at byte 106 is: (small to...)
```
So, I imagine my issue is something related with that `Dump.dump`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2748893458
@hdalsania @JeremyYao So this PR does only the left over data message as a popup. It does not have a popup for the other two errors that @JeremyYao mentioned, those would want to be separate issues but could probably be 1 PR that fixes both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on code in PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#discussion_r2007917494
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Utils.scala:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.debugger.dap
+
+import java.nio.charset.StandardCharsets
+
+/** Data Left Over Utils */
+object DataLeftOverUtils {
Review Comment:
@stevedlawrence That is what I am assuming since the message is correct and when using `println` it can display the message perfectly fine. But I figure something else is wrong because that isn't the same thing daffodil-cli shows, daffodil-cli doesn't have the `␀␀`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2755332057
I've figured out that there's some kind of timing/async issue. The issue can be reproduced by replacing `Dump.dump(...)` with `Thread.sleep(5000)`. So I don't think this has anything to do with `Dump.dump` except that it adds enough overhead to triggeri some timing issues.
I'm guessing that after the parse completes something starts shutting down or closing the DAP connection, but that doesn't happen immediately, which allows enough time for a simple data left over message to be sent but not a more complex one like Dump.dump.
I'm not familiar enough with the core of VS Code extension to know how this all works, but it looks like it makes use of cats-effects. That library is pretty heavily asynchronus so an timing/async bug isn't too surprising.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "rthomas320 (via GitHub)" <gi...@apache.org>.
rthomas320 commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2754428246
I pulled in the code for PR #1204 Left over data message. I am not seeing the left over data message on Windows. 10. Also the Omega Edit server window is no longer closing after debugging finishes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2755344798
@stevedlawrence That makes a lot of sense, so we basically need to make it so that basically the program allows for the `Dump.dump(...)` to fully complete before trying to close out? Now, from what you are saying its going to be something with `cats-effects` and not the DataDumper itself right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2755376703
@stevedlawrence found the fix. I had the reorder methods the parse methods, so it needed to be this:
```scala
val parse =
IO.interruptibleMany(
parseResult
// WARNING: parse doesn't close the OutputStream, so closed below
).ensureOr(res => new Parse.Exception(res.getDiagnostics.toList))(res => !res.isError)
.void
.flatMap { _ =>
val leftOverBits = (dataSize - (loc.bytePos1b - 1)) * 8
if (leftOverBits > 0) {
val message = DataLeftOverUtils.getMessage(dataFilePath, loc.bitPos1b, loc.bytePos1b, leftOverBits)
logger.error(message) *> dapEvents.send(
DataLeftOverEvent(loc.bitPos1b, loc.bytePos1b, leftOverBits, message)
) *> IO.unit
} else {
IO.unit
}
}
.guarantee(IO(os.close) *> done.set(true))
```
So I needed to flip where `.flatMap` and `.guarantee` were and everything seems to work!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on code in PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#discussion_r2007903396
##########
debugger/src/main/scala/org.apache.daffodil.debugger.dap/Utils.scala:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.debugger.dap
+
+import java.nio.charset.StandardCharsets
+
+/** Data Left Over Utils */
+object DataLeftOverUtils {
Review Comment:
@stevedlawrence So I did find its something with the dump string, when I print it out it looks like this:
```bash
Left over data (UTF-8) starting at byte 106 is: (small␣to␀␀...)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "JeremyYao (via GitHub)" <gi...@apache.org>.
JeremyYao commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2743595140
Performed tests as outlined in the ticket.
for first test case.
For parse error test case, seems to show the Parse Error in the debug console, but pop-up on the bottom right doesn't show
Test case for bad schema definition error shows this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add left over data message to debugger and extension [daffodil-vscode]
Posted by "hdalsania (via GitHub)" <gi...@apache.org>.
hdalsania commented on PR #1204:
URL: https://github.com/apache/daffodil-vscode/pull/1204#issuecomment-2749050769
> @hdalsania @JeremyYao So this PR does only the left over data message as a popup. It does not have a popup for the other two errors that @JeremyYao mentioned, those would want to be separate issues but could probably be 1 PR that fixes both.
Make sense. We will create a new issue for this and track that separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org