You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by "Goden Yao (JIRA)" <ji...@apache.org> on 2016/01/04 22:17:39 UTC
[jira] [Updated] (HAWQ-315) Invalid Byte Sequence Error when
loading a large(100MB+) csv file
[ https://issues.apache.org/jira/browse/HAWQ-315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Goden Yao updated HAWQ-315:
---------------------------
Description:
This bug occurs when copying or reading a large csv file. The reproducible file we tried is 100MB+ so cannot be uploaded to JIRA.
*repro steps*
The large CSV file size needs to be over 100MB at least.
The pattern in the csv file should contain the following:
{code:actionscript}
..., "dummy data text1
dummy data text2,
dummy data text3,
dummy data text4"
{code}
basically , a long text broken into multiple lines but within quotes.
This doesn't cause issue in a smaller size file though.
{code:SQL}
DROP TABLE IF EXISTS <_test table name_>;
CREATE TABLE <_test table name_>
(
<_define test table schema_>
...
);
COPY <_test table name_> FROM '<_csv file path_>'
DELIMITER ','
NULL ''
ESCAPE '"'
CSV QUOTE '"'
LOG ERRORS INTO <_error reject table name_>
SEGMENT REJECT LIMIT 10 rows
;
{code}
*Errors*
Error in first line with quoted data:
{code}
DEBUG5: invalid byte sequence for encoding "UTF8": 0x00
HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
CONTEXT: COPY addresses_heap, line 604932
{code}
Error in second line with quoted data: this is due to wrong formatting (as the first half line within the quotes was mishandled.
{code}
DEBUG5: missing data for column "sourceid"
CONTEXT: COPY addresses_heap, line 604933: "...
{code}
was:
This bug occurs when copying or reading a large csv file. The reproducible file we tried is 100MB+ so cannot be uploaded to JIRA.
*repro steps*
The large CSV file size needs to be over 100MB at least.
The pattern in the csv file should contain the following:
{code:csv}
..., "dummy data text1
dummy data text2,
dummy data text3,
dummy data text4"
{code}
basically , a long text broken into multiple lines but within quotes.
This doesn't cause issue in a smaller size file though.
{code:SQL}
DROP TABLE IF EXISTS <_test table name_>;
CREATE TABLE <_test table name_>
(
<_define test table schema_>
...
);
COPY <_test table name_> FROM '<_csv file path_>'
DELIMITER ','
NULL ''
ESCAPE '"'
CSV QUOTE '"'
LOG ERRORS INTO <_error reject table name_>
SEGMENT REJECT LIMIT 10 rows
;
{code}
*Errors*
Error in first line with quoted data:
{code}
DEBUG5: invalid byte sequence for encoding "UTF8": 0x00
HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
CONTEXT: COPY addresses_heap, line 604932
{code}
Error in second line with quoted data: this is due to wrong formatting (as the first half line within the quotes was mishandled.
{code}
DEBUG5: missing data for column "sourceid"
CONTEXT: COPY addresses_heap, line 604933: "...
{code}
> Invalid Byte Sequence Error when loading a large(100MB+) csv file
> -----------------------------------------------------------------
>
> Key: HAWQ-315
> URL: https://issues.apache.org/jira/browse/HAWQ-315
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Core
> Reporter: Goden Yao
> Assignee: Lei Chang
>
> This bug occurs when copying or reading a large csv file. The reproducible file we tried is 100MB+ so cannot be uploaded to JIRA.
> *repro steps*
> The large CSV file size needs to be over 100MB at least.
> The pattern in the csv file should contain the following:
> {code:actionscript}
> ..., "dummy data text1
> dummy data text2,
> dummy data text3,
> dummy data text4"
> {code}
> basically , a long text broken into multiple lines but within quotes.
> This doesn't cause issue in a smaller size file though.
> {code:SQL}
> DROP TABLE IF EXISTS <_test table name_>;
> CREATE TABLE <_test table name_>
> (
> <_define test table schema_>
> ...
> );
> COPY <_test table name_> FROM '<_csv file path_>'
> DELIMITER ','
> NULL ''
> ESCAPE '"'
> CSV QUOTE '"'
> LOG ERRORS INTO <_error reject table name_>
> SEGMENT REJECT LIMIT 10 rows
> ;
> {code}
> *Errors*
> Error in first line with quoted data:
> {code}
> DEBUG5: invalid byte sequence for encoding "UTF8": 0x00
> HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
> CONTEXT: COPY addresses_heap, line 604932
> {code}
> Error in second line with quoted data: this is due to wrong formatting (as the first half line within the quotes was mishandled.
> {code}
> DEBUG5: missing data for column "sourceid"
> CONTEXT: COPY addresses_heap, line 604933: "...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)