fastest way-sqoop import from teradata

["Sandipan.Ghosh" <Sa...@target.com>]

Hi All,

What is the fastest way of import a single Teradata table from Teradata to HDFS using sqoop?

I have a 200 gb table which is taking around 540 hours to transfer. I am in cloudera CDH.

I am using

sqoop import \
-D sqoop.teradata.import.use.temporary.table=false \
--connect jdbc:teradata://10.67.192.180/DATABASE=INOVBIDT,LOGMECH=LDAP \
--username z070061 --password ******* \
--compress \
--fields-terminated-by '|' \
--table inovbidt.Item_Loc_Day_Oh_Inv_41 \
--split-by GREG_D \
--target-dir /innovation/distribution_transportation/temp/Item_Loc_Day_Oh_Inv_str_41_2012 \
--mapreduce-job-name Item_Loc_Day_Oh_Inv_str_41_2012

Thanks
Sandipan

Re: fastest way-sqoop import from teradata

[Jarek Jarcec Cecho <ja...@apache.org>]

Hi Sandipan,
there are highly optimized Teradata connectors out there and I would suggest you to use them for transferring data between Hadoop and Teradata.

Since you are using CDH, you might be interested in the Cloudera one:

http://www.cloudera.com/content/support/en/downloads/download-components/download-products/downloads-listing/connectors/teradata.html

Jarcec

On Wed, May 07, 2014 at 08:55:26AM +0000, Sandipan.Ghosh wrote:
> Hi All,
> 
> What is the fastest way of import a single Teradata table from Teradata to HDFS using sqoop?
> 
> I have a 200 gb table which is taking around 540 hours to transfer. I am in cloudera CDH.
> 
> I am using
> 
> sqoop import \
> -D sqoop.teradata.import.use.temporary.table=false \
> --connect jdbc:teradata://10.67.192.180/DATABASE=INOVBIDT,LOGMECH=LDAP \
> --username z070061 --password ******* \
> --compress \
> --fields-terminated-by '|' \
> --table inovbidt.Item_Loc_Day_Oh_Inv_41 \
> --split-by GREG_D \
> --target-dir /innovation/distribution_transportation/temp/Item_Loc_Day_Oh_Inv_str_41_2012 \
> --mapreduce-job-name Item_Loc_Day_Oh_Inv_str_41_2012
> 
> Thanks
> Sandipan