[jira] [Commented] (FLINK-8331) FieldParsers do not correctly set EMPT_COLUMN error state

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-8331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16311499#comment-16311499 ] 

ASF GitHub Bot commented on FLINK-8331:
---------------------------------------

Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5218#discussion_r159678062
  
    --- Diff: flink-core/src/test/java/org/apache/flink/types/parser/FieldParserTest.java ---
    @@ -43,4 +46,100 @@ public void testEndsWithDelimiter() throws Exception {
     		assertFalse(FieldParser.endsWithDelimiter(bytes, 3, delim));
     	}
     
    -}
    \ No newline at end of file
    +	@Test
    +	public void testNextStringEndPos() throws Exception {
    +
    +		FieldParser parser = new TestFieldParser<String>();
    +		// single-char delimiter
    +		byte[] singleCharDelim = "|".getBytes(ConfigConstants.DEFAULT_CHARSET);
    +
    +		byte[] bytes1 = "a|".getBytes(ConfigConstants.DEFAULT_CHARSET);
    +		assertEquals(1, parser.nextStringEndPos(bytes1, 0, bytes1.length, singleCharDelim));
    +		assertEquals(-1, parser.nextStringEndPos(bytes1, 1, bytes1.length, singleCharDelim));
    +		assertEquals(ParseErrorState.EMPTY_COLUMN, parser.getErrorState());
    +
    +		parser.resetParserState();
    --- End diff --
    
    please add 
    
    ```
    parser.resetParserState();
    assertEquals(-1, parser.nextStringEndPos(bytes1, 2, bytes1.length, singleCharDelim));
    assertEquals(ParseErrorState.EMPTY_COLUMN, parser.getErrorState());
    ```


> FieldParsers do not correctly set EMPT_COLUMN error state
> ---------------------------------------------------------
>
>                 Key: FLINK-8331
>                 URL: https://issues.apache.org/jira/browse/FLINK-8331
>             Project: Flink
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.5.0, 1.4.1
>            Reporter: Fabian Hueske
>            Assignee: sunjincheng
>
> Some {{FieldParser}} do not correctly set the EMPTY_COLUMN error state if a field is empty.
> Instead, they try to parse the field value from an empty String which fails, e.g., in case of the {{DoubleParser}} with a {{NumberFormatException}}.
> The {{RowCsvInputFormat}} has a flag to interpret empty fields as {{null}} values. The implementation requires that all {{FieldParser}} correctly return the EMPTY_COLUMN error state in case of an empty field.
> Affected {{FieldParser}}:
> - BigDecParser
> - BigIntParser
> - DoubleParser
> - FloatParser
> - SqlDateParser
> - SqlTimeParser
> - SqlTimestampParser



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)