[jira] [Commented] (DAFFODIL-935) DFDL encodingErrorPolicy='error' feature

["Mike Beckerle (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DAFFODIL-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256079#comment-17256079 ] 

Mike Beckerle commented on DAFFODIL-935:
----------------------------------------

Cranking up priority of this, because it causes first-order user-experience pain.

New users run into this. E.g., on the CSV schema, which is portable, the schema has dfdl:encodingErrorPolicy='error', and that causes every element of the schema that has text represtation (which is all of them) to each issue a warning.

The DFDL Spec section on encodingErrorPolicy='error' is  section {color:#000000}11.2.1.1 (of latest working draft. {color}here: [https://opengridforum.github.io/DFDL/working-drafts/gwdrp-dfdl-v1.0.5-r35.htm#_Toc54264422] )

This specifies specific kinds of decode errors that must be detected. It also specifies some kinds of decode errors that may not be detected.

A subtle case is regex matching, whether for lengthKind 'pattern' or assert/discriminator with testKind 'pattern'.

In those cases, any decode error that occurs as part of examining data to determine the regex match (or not), can be raised as a processing error.

However, daffodil's regex matching algorithm does the following:
 # Create and decode text into a buffer of size N.
 # Try to match the regex against the buffer. If hit-end, then double the buffer size, re-fill it with decoded text and try again.
 # Repeat until the match does not 'hit-end', or the size limit (a tunable) is reached.

So it is possible for this to happen: had the buffer been smaller, the match might have been obtained without any decode error being received, but just due to the accident of the buffer re-sizing to a bigger size, a decode-error occurred when it was filled in.

When dfdl:encodingErrorPolicy is 'error' then I believe the right behavior is that a decode-error occurring when filling in the regex match buffer with text, should stop the filling of the buffer, and the size of the buffer should be limited to the successfully decoded characters up until the decode-error occurred. The decode error, and its position in the data stream, should be saved.

Then the match should be attempted. If hit-end occurs, then a decode error (processing error) should be raised with the details about the decode error that were saved.

If a match occurs without hit-end, or no-match without hit-end, then the decode-error should be discarded.

This change has the potential to change the behavior of existing DFDL schemas that are using dfdl:encodingErrorPolicy='error', but are using Daffodil's current fake implementation of this (which behaves exactly like 'replace' policy). In particular, a decoding error may be detected now where a replacement character was created before. The workaround for this is of course to change the dfdl:encodingErrorPolicy to 'replace' for that schema. 

 

 

> DFDL encodingErrorPolicy='error' feature
> ----------------------------------------
>
>                 Key: DAFFODIL-935
>                 URL: https://issues.apache.org/jira/browse/DAFFODIL-935
>             Project: Daffodil
>          Issue Type: New Feature
>          Components: Back End, DFDL Language
>    Affects Versions: s13
>            Reporter: Mike Beckerle
>            Priority: Minor
>
> 0.13.0 has encodingErrorPolicy="replace". (DFDL-893). This is to implement the much more challenging precise-error feature. This creates performance challenges because one cannot buffer-ahead character conversion from bytes in the obvious ways that say, java.io.BufferedReader does.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)