You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Eric <en...@gmail.com> on 2015/07/29 03:25:46 UTC

Can't export data - Expected delimiter at position

Hi here,

I'm trying to export some data from amazon S3 to postgresql. The data are
coming from redshift and the unload command.

It works well with 20millions entries. However , with 800millions, I got
some errors right after the map-reduce job starts. I also tried with
different delimiters. The data are escaped with the backslash character,
and the fields are enclosed by double-quotes.

My command is:
sqoop export  --connect jdbc:postgresql://uri/db --username [] --table []
--export-dir s3://[]  --password '7dzeqwYb?WaqKMGPz8NA(y'
--fields-terminated-by ',' --enclosed-by '\"' --null-string ""
--null-non-string "" --escaped-by \\

I was wondering if there was a way to fine the entries causing this? I
tried to find in all the logs and couldn't find anything.
Do I miss anything? Is there a proper way to solve this? Thank you !


The output :
Error: java.io.IOException: Can't export data, please check failed map task
logs
 at
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)

    at
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)

    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:152)

    at
org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)

    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:773)

    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:415)

    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:170)

Caused by: com.cloudera.sqoop.lib.RecordParser$ParseError: Expected
delimiter at position 235

    at org.apache.sqoop.lib.RecordParser.parseRecord(RecordParser.java:319)

    at org.apache.sqoop.lib.RecordParser.parseRecord(RecordParser.java:108)

    at org.apache.sqoop.lib.RecordParser.parseRecord(RecordParser.java:125)

    at updates_backfill.parse(updates_backfill.java:1498)

    at
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)

Re: Can't export data - Expected delimiter at position

Posted by Abraham Elmahrek <ab...@cloudera.com>.
Hey man,

Try adding --verbose. Normally Sqoop will let you know the records that are
failing.

-Abe

On Tue, Jul 28, 2015 at 6:25 PM, Eric <en...@gmail.com> wrote:

> Hi here,
>
> I'm trying to export some data from amazon S3 to postgresql. The data are
> coming from redshift and the unload command.
>
> It works well with 20millions entries. However , with 800millions, I got
> some errors right after the map-reduce job starts. I also tried with
> different delimiters. The data are escaped with the backslash character,
> and the fields are enclosed by double-quotes.
>
> My command is:
> sqoop export  --connect jdbc:postgresql://uri/db --username [] --table []
> --export-dir s3://[]  --password '7dzeqwYb?WaqKMGPz8NA(y'
> --fields-terminated-by ',' --enclosed-by '\"' --null-string ""
> --null-non-string "" --escaped-by \\
>
> I was wondering if there was a way to fine the entries causing this? I
> tried to find in all the logs and couldn't find anything.
> Do I miss anything? Is there a proper way to solve this? Thank you !
>
>
> The output :
> Error: java.io.IOException: Can't export data, please check failed map
> task logs
>  at
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
>
>     at
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
>
>     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:152)
>
>     at
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>
>     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:773)
>
>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>
>     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
>
>     at java.security.AccessController.doPrivileged(Native Method)
>
>     at javax.security.auth.Subject.doAs(Subject.java:415)
>
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:170)
>
> Caused by: com.cloudera.sqoop.lib.RecordParser$ParseError: Expected
> delimiter at position 235
>
>     at org.apache.sqoop.lib.RecordParser.parseRecord(RecordParser.java:319)
>
>     at org.apache.sqoop.lib.RecordParser.parseRecord(RecordParser.java:108)
>
>     at org.apache.sqoop.lib.RecordParser.parseRecord(RecordParser.java:125)
>
>     at updates_backfill.parse(updates_backfill.java:1498)
>
>     at
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
>
>