[jira] [Created] (PHOENIX-7267) CsvBulkLoadTool fails for a bad record with "(startline 1) EOF reached before encapsulated token finished"

["Nihal Jain (Jira)" <ji...@apache.org>]

Nihal Jain created PHOENIX-7267:
-----------------------------------

             Summary: CsvBulkLoadTool fails for a bad record with "(startline 1) EOF reached before encapsulated token finished"
                 Key: PHOENIX-7267
                 URL: https://issues.apache.org/jira/browse/PHOENIX-7267
             Project: Phoenix
          Issue Type: Bug
    Affects Versions: 5.1.3, 5.2.0, 5.3.0
            Reporter: Nihal Jain
            Assignee: Nihal Jain


We are trying to load data where there are few bad record for some files due to which mappers fail and hence the entire job fail with following error:
{code:java}
Error: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: (startline 1) EOF reached before encapsulated token finished
	at org.apache.phoenix.mapreduce.FormatToBytesWritableMapper.map(FormatToBytesWritableMapper.java:206)
	at org.apache.phoenix.mapreduce.FormatToBytesWritableMapper.map(FormatToBytesWritableMapper.java:77)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.lang.RuntimeException: java.io.IOException: (startline 1) EOF reached before encapsulated token finished
	at org.apache.commons.csv.CSVParser$1.getNextRecord(CSVParser.java:398)
	at org.apache.commons.csv.CSVParser$1.hasNext(CSVParser.java:407)
	at org.apache.phoenix.thirdparty.com.google.common.collect.Iterators.getNext(Iterators.java:895)
	at org.apache.phoenix.thirdparty.com.google.common.collect.Iterables.getFirst(Iterables.java:827)
	at org.apache.phoenix.mapreduce.CsvToKeyValueMapper$CsvLineParser.parse(CsvToKeyValueMapper.java:109)
	at org.apache.phoenix.mapreduce.CsvToKeyValueMapper$CsvLineParser.parse(CsvToKeyValueMapper.java:91)
	at org.apache.phoenix.mapreduce.FormatToBytesWritableMapper.map(FormatToBytesWritableMapper.java:164)
	... 9 more
Caused by: java.io.IOException: (startline 1) EOF reached before encapsulated token finished
	at org.apache.commons.csv.Lexer.parseEncapsulatedToken(Lexer.java:282)
	at org.apache.commons.csv.Lexer.nextToken(Lexer.java:152)
	at org.apache.commons.csv.CSVParser.nextRecord(CSVParser.java:450)
	at org.apache.commons.csv.CSVParser$1.getNextRecord(CSVParser.java:395)
	... 15 more {code}
I have figured out there is code in commons-csv which throws a RuntimeException when it fails to parse are record which is not handled by phoenix as we only catch IOException. 

See [https://github.com/apache/commons-csv/blob/rel/commons-csv-1.0/src/main/java/org/apache/commons/csv/CSVParser.java#L398] 

Also see [https://github.com/apache/phoenix/blob/master/phoenix-core-server/src/main/java/org/apache/phoenix/mapreduce/FormatToBytesWritableMapper.java#L167]

 

This is undesired, in worst case the job should just skip the failed record than the whole job. Note we are passing --ignore-errors.

This bug is to fix this behavior.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)