You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by "lrbarber (via GitHub)" <gi...@apache.org> on 2024/02/27 16:35:12 UTC
[I] No warning given when data left over after parse. [daffodil-vscode]
lrbarber opened a new issue, #974:
URL: https://github.com/apache/daffodil-vscode/issues/974
When running daffodil from the command line, if a schema parse completes, but does not parse the entire data file, a warning message about there being left over data is issued.
When this occurs inside Visual Studio, there is no such warning message.
There may be other similar messages that are not being presented by the debugger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No warning given when data left over after parse. [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on issue #974:
URL: https://github.com/apache/daffodil-vscode/issues/974#issuecomment-2715511078
I don't think I know enough about VS Code to know what the best way to handle the message. I imagine if it was passed back to the extension then you could do some fancier things like provide an option to open the data editor with the cursor pointed to where the parse stopped. But whether or not thats really needed is up to you.
And since you have the data editor evailable, outputting following byte information seems less important. A user can also pop open the editor and jump to the bit position and look at it that way.
I'll add one thing, that if you do log it instead of passing it back to the extension, you might do it as an error. Daffodil used to treat left over data as a warning. But most users expect a schema to consume the entire data, and wanted a error when that doesn't happen, so we now output it as an error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No warning given when data left over after parse. [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on issue #974:
URL: https://github.com/apache/daffodil-vscode/issues/974#issuecomment-2715435727
@stevedlawrence Do you think it would be better to have this message be an `error` or `warn` in the debugger log or should the debugger passes the message to the extension and then the extension displays the message? Currently when testing it I was making it a warn message inside of the debuggers log. So far it looks something like this:
```bash
2025-03-11 15:07:09,410 [io-compute-1] WARN o.a.d.d.d.Parse - Left over data. Consumed 840 bit(s), Left over data starts at byte 106.
```
The reason for not currently displaying the leftOverBites is because the way I could get it to work, it was returning a different value then daffodil-cli. I also didn't currently display the next byte because it seemed like the way Daffodil did it won't work in my situation because it seems we are using a more generic `InputStream` for the data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No warning given when data left over after parse. [daffodil-vscode]
Posted by "lrbarber (via GitHub)" <gi...@apache.org>.
lrbarber commented on issue #974:
URL: https://github.com/apache/daffodil-vscode/issues/974#issuecomment-2023778748
Example of error message:
```
[error] Left over data. Consumed 1280 bit(s) with at least 64 bit(s) remaining.
Left over data (Hex) starting at byte 161 is: (0x3ff278eabbeb1a00...)
Left over data (UTF-8) starting at byte 161 is: (?�x��␚␀...)
```
Other types of messages are also not being displayed:
```
[warn] Schema Definition Warning: In expression Message_Identifier, result type (unsignedInt) should be manually cast to the expected type (NonEmptyString) with the appropriate constructor.Performing deprecated automatic conversion. (id: deprecatedExpressionResultCoercion)
Schema context: choice[1] Location line 41 column 18 in file:/vagrant/test.dfdl.xsd
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No warning given when data left over after parse. [daffodil-vscode]
Posted by "JeremyYao (via GitHub)" <gi...@apache.org>.
JeremyYao commented on issue #974:
URL: https://github.com/apache/daffodil-vscode/issues/974#issuecomment-2698479017
## More Test Cases
Follow instructions as outlined in https://github.com/apache/daffodil-vscode/issues/974#issuecomment-2698420246, but copy the corresponding DFDL schemas and input files.
The corresponding DFDL schemas and input files are contained in the following zip file. Note that you will need to include all of the files in the .zip as the specified DFDL files depend on the other files in there.
[extra_test_cases.zip](https://github.com/user-attachments/files/19075471/extra_test_cases.zip)
### Parse error
Powershell Output
```PowerShell
PS C:\Users\jeremy.yao\repos\daffodilWorkspace\PPT Examples\974Test> daffodil parse --schema .\test.dfdl_parse_err.dfdl.xsd .\test_parse_err.txt
[error] Parse Error: Unable to parse xs:integer from text: According to all known laws of aviation, there is no way a bee should be able to fly
Schema context: label Location line 16 column 18 in test.dfdl_parse_err.dfdl.xsd
Data location was preceding byte 84
```
- DFDL Schema: test.dfdl_parse_err.dfdl.xsd
- Test file: test_parse_err.txt
### Schema Definition Error
```PowerShell
PS C:\Users\jeremy.yao\repos\daffodilWorkspace\PPT Examples\974Test> daffodil parse --schema .\test.dfdl_parse_err.dfdl.xsd .\test_parse_err.txt
[error] Parse Error: Unable to parse xs:integer from text: According to all known laws of aviation, there is no way a bee should be able to fly
Schema context: label Location line 16 column 18 in test.dfdl_parse_err.dfdl.xsd
Data location was preceding byte 84
PS C:\Users\jeremy.yao\repos\daffodilWorkspace\PPT Examples\974Test> daffodil parse --schema .\bad_dfdl.dfdl.xsd .\test_parse_err.txt
[error] Schema Definition Error: Type integer cannot have lengthKind='implicit' when representation='text'
Schema context: label Location line 16 column 18 in bad_dfdl.dfdl.xsd
```
- DFDL Schema: test.dfdl_parse_err.dfdl.xsd
- Test file: test_parse_err.txt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No warning given when data left over after parse. [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell closed issue #974: No warning given when data left over after parse.
URL: https://github.com/apache/daffodil-vscode/issues/974
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No warning given when data left over after parse. [daffodil-vscode]
Posted by "JeremyYao (via GitHub)" <gi...@apache.org>.
JeremyYao commented on issue #974:
URL: https://github.com/apache/daffodil-vscode/issues/974#issuecomment-2698420246
### How to Replicate (Left over data message)
1. Download the following zip file and use the contents of the file as your sampleWorkspace: [sampleWorkspace_974_left_over_data.zip](https://github.com/user-attachments/files/19075082/sampleWorkspace_974_left_over_data.zip)
2. Once extracted and mounted as your sampleWorkspace, navigate to the folder in a terminal and enter `daffodil parse --schema .\test.dfdl.xsd .\test.txt`. You should notice that daffodil will output the following error message.
```PowerShell
PS C:\Users\jeremy.yao\repos\sampleWorkspace> daffodil parse --schema .\test.dfdl.xsd .\test.txt
<?xml version="1.0" encoding="UTF-8"?>
<input>
<label>According to all known laws of aviation, there is no way a bee should be able to fly</label>
<colon>:</colon>
<message>
Its wings are too </message>
</input>
[error] Left over data. Consumed 840 bit(s) with at least 405840 bit(s) remaining.
Left over data (Hex) starting at byte 106 is: (0x736d616c6c20746f...)
Left over data (UTF-8) starting at byte 106 is: (small?to...)
```
4. Run the 974Test_1 run and debug configuration. Note: This configuration uses test.dfdl.xsd for its DFDL schema and test.txt as its input file.

5. Click on continue (or step through the file if you want)

6. Notice that after completing the debugging, no warning message pops up

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No warning given when data left over after parse. [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell commented on issue #974:
URL: https://github.com/apache/daffodil-vscode/issues/974#issuecomment-2711461872
@stevedlawrence @mbeckerle I have been working on this issue and can't quite figure it out. I was looking at the file `daffodil-cli/src/main/scala/org/apache/daffodil/cli/Main.scala` around lines 1265 to 1371 in the Daffodil repository. However, we use `sapi` for most things in the debugger so I am don't have access to certain things like:
```scala
parseResult.resultState // value resultState is not a member of org.apache.daffodil.sapi.ParseResult
```
then that result state's attributes. Do you guys have any tips on what I should be looking at that will allow me to add a message in the VSCode debugger that is similar to the CLI debugger?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No warning given when data left over after parse. [daffodil-vscode]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on issue #974:
URL: https://github.com/apache/daffodil-vscode/issues/974#issuecomment-2711515697
There is kindof two ways to do it. One option is to ask if the InputSourceDataInputStream has any data available, for example:
```scala
val isdis = new InputSourceDataInputStream(data)
val pr = dp.parse(isdis, infosetOutputter)
if (!pr.isError) {
if (isdis.hasData) {
message("At least one bit of left over data found")
}
}
```
Note that if your data is coming from an InputStream, then hasData will block until it either hits EOF or reads at least one byte. This doesn't tell you how much left over data there was or what the following data is in the stream, which can sometimes be a useful diagnostic. Unfortunately, the public Daffodil API doesn't help with this too much, which is why Main.scala does things a bit more complicatedly.
What you could do, and essentially what Daffodil does, is get the bit or byte position (1-based) from the parse result, e.g.
```scala
val endingBitPosition = pr.location.bitPos1b
```
and compare that against how much data you provided to Daffodil. That's not too hard if your data is something with a fixed size like a `File` or `Array[Byte]`, but it's a bit tricker if your input is a generic `InputStream`. This has the added benefit that if your input is an File or Array, you can just jump to that location and output the next handful of bytes if you want a more verbose message.
Note that even if you go with the first approach, outputing the ending bit position might be useful information to include in the message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No warning given when data left over after parse. [daffodil-vscode]
Posted by "shanedell (via GitHub)" <gi...@apache.org>.
shanedell closed issue #974: No warning given when data left over after parse.
URL: https://github.com/apache/daffodil-vscode/issues/974
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] No warning given when data left over after parse. [daffodil-vscode]
Posted by "JeremyYao (via GitHub)" <gi...@apache.org>.
JeremyYao commented on issue #974:
URL: https://github.com/apache/daffodil-vscode/issues/974#issuecomment-2704433095
@hdalsania Here is what I'm seeing on the problems tab.
### Scenario 1: Start debug, step over a few times, and continue
Just starting the debugging session

A few step overs

Continued till the end

### Scenario 2: Start debug and continue

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org