Re: hive error: "Too many bytes before delimiter: 2147483648"

[Shawn Weeks <sw...@weeksconsulting.us>]

That looks like you’ve encountered a file with no delimiter as that’s near the max size for an array or string. Also I don’t think you can terminate fields with a line feed as that’s the hard coded row delimiter.

Thanks
Shawn

From: xuanhuang <18...@163.com>
Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
Date: Saturday, November 30, 2019 at 1:40 AM
To: "user@hive.apache.org" <us...@hive.apache.org>
Subject: hive error: "Too many bytes before delimiter: 2147483648"


Hello all,
I encountered a problem while using hive, please ask you. The following is the specific situation.
Platform: hive on spark
Error: java.io.IOException: Too many bytes before delimiter: 2147483648
Description: When using small files of the same format for testing, there is no problem; but for all data (about 140G), the above error is reported.
Table creation statement:
DROP TABLE IF EXISTS test;
CREATE EXTERNAL TABLE test (
     json string
     )
     ROW FORMAT DELIMITED
     FIELDS TERMINATED BY '\ r \ n'
     STORED AS TEXTFILE

LOCATION 'hdfs: / userdata / test';

The big data platform I use is TDH Community Edition 5.0.2.
Any help is much appreciated!