You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Benyi Wang <be...@gmail.com> on 2014/06/18 03:09:51 UTC
Sqoop hangs when import from Teradata
I have trouble import Teradata table into HDFS:
1. Installed CDH using cloudera manager on CentOS 6.
2. Installed Sqoop-Teradata-Connector 1.2.5c5 using parcel.
3. Install Sqoop Client service, and deploy the configuration file.
4. Downloaded teradata jdbc driver v15.0.0.15 and v13.10.00.36, and
extracted the jars in ~/apps/terajdbc.
5. export
HADOOP_CLASSPATH=~/apps/terajdbc/terajdbc4.jar:~/apps/terajdbc/tdgssconfig.jar
6. sqoop import --connect jdbc:teradata://teraserver/DATABASE=MYDB
--username bewang -P --table action --num-mappers 1, then it just hangs
after creating jar. The table 'action' is pretty small, only 161 rows.
14/06/17 18:05:08 INFO orm.CompilationManager: Writing jar file:
/tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.jar
14/06/17 18:05:08 DEBUG orm.CompilationManager: Scanning for .class files
in directory: /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef
14/06/17 18:05:08 DEBUG orm.CompilationManager: Got classfile:
/tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.class
-> action.class
14/06/17 18:05:08 DEBUG orm.CompilationManager: Finished writing jar file
/tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.jar
14/06/17 18:05:08 INFO teradata.TeradataManager: Beginning Teradata table
based import
1. I could successfully run "sqoop eval" to dump the table without
problem.
What's wrong with my method?
Re: Sqoop hangs when import from Teradata
Posted by Abraham Elmahrek <ab...@cloudera.com>.
Thanks for the info. Also, thanks for getting back to the mailing list and
providing your solution.
On Wed, Jun 18, 2014 at 2:49 PM, Benyi Wang <be...@gmail.com> wrote:
> I tried --verbose before, and no any affect. I waited for it until the
> error message is shown.
>
>
> On Wed, Jun 18, 2014 at 2:01 PM, Abraham Elmahrek <ab...@cloudera.com>
> wrote:
>
>> Hey there,
>>
>> Glad every thing is working... and sorry it took so long. Did you add the
>> --verbose flag to the beginning of the import statement? Or did you need to
>> wait for the job to finish?
>>
>> -Abe
>>
>>
>> On Wed, Jun 18, 2014 at 1:56 PM, Benyi Wang <be...@gmail.com>
>> wrote:
>>
>>> It turns out that I have to provide "--split-by actionId". It took
>>> 53mins to print out the exception in my system.
>>>
>>> 14/06/18 12:04:33 INFO teradata.TeradataManagerFactory: Loaded connector
>>> factory for 'Cloudera Connector Powered by Teradata' on version 1.2c5
>>> 14/06/18 12:04:33 INFO manager.SqlManager: Using default fetchSize of
>>> 1000
>>> 14/06/18 12:04:33 INFO options.ExtraOptions: Parsing extra arguments
>>> 14/06/18 12:04:33 INFO options.ExtraOptions: Loaded extra option for
>>> batch size 1000
>>> 14/06/18 12:04:33 INFO options.OptionsCompatibility: Checking options
>>> compatibility
>>> 14/06/18 12:04:35 INFO tool.CodeGenTool: Beginning code generation
>>> 14/06/18 12:04:35 INFO teradata.TeradataManager: Converting table import
>>> to query: SELECT * FROM "action"
>>> 14/06/18 12:04:36 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
>>> /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
>>> Note:
>>> /tmp/sqoop-bwang/compile/405654670de634debca58668abd0d1f4/action.java uses
>>> or overrides a deprecated API.
>>> Note: Recompile with -Xlint:deprecation for details.
>>> 14/06/18 12:04:40 INFO orm.CompilationManager: Writing jar file:
>>> /tmp/sqoop-bwang/compile/405654670de634debca58668abd0d1f4/action.jar
>>> 14/06/18 12:04:40 INFO teradata.TeradataManager: Beginning Teradata
>>> table based import
>>> 14/06/18 12:57:25 ERROR sqoop.Sqoop: Got exception running Sqoop:
>>> java.lang.RuntimeException: Table action does not have primary key. Please
>>> use --split-by argument if applicable
>>> java.lang.RuntimeException: Table action does not have primary key.
>>> Please use --split-by argument if applicable
>>> at
>>> com.cloudera.connector.teradata.TeradataManager.getPrimaryKey(TeradataManager.java:175)
>>> at
>>> org.apache.sqoop.manager.SqlManager.getSplitColumn(SqlManager.java:546)
>>> at
>>> com.cloudera.connector.teradata.TeradataManager.importTable(TeradataManager.java:273)
>>> at
>>> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
>>> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:506)
>>> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>>> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>>>
>>>
>>>
>>> On Tue, Jun 17, 2014 at 6:09 PM, Benyi Wang <be...@gmail.com>
>>> wrote:
>>>
>>>> I have trouble import Teradata table into HDFS:
>>>>
>>>> 1. Installed CDH using cloudera manager on CentOS 6.
>>>> 2. Installed Sqoop-Teradata-Connector 1.2.5c5 using parcel.
>>>> 3. Install Sqoop Client service, and deploy the configuration file.
>>>> 4. Downloaded teradata jdbc driver v15.0.0.15 and v13.10.00.36, and
>>>> extracted the jars in ~/apps/terajdbc.
>>>> 5. export
>>>> HADOOP_CLASSPATH=~/apps/terajdbc/terajdbc4.jar:~/apps/terajdbc/tdgssconfig.jar
>>>> 6. sqoop import --connect jdbc:teradata://teraserver/DATABASE=MYDB
>>>> --username bewang -P --table action --num-mappers 1, then it just hangs
>>>> after creating jar. The table 'action' is pretty small, only 161 rows.
>>>>
>>>> 14/06/17 18:05:08 INFO orm.CompilationManager: Writing jar file:
>>>> /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.jar
>>>> 14/06/17 18:05:08 DEBUG orm.CompilationManager: Scanning for .class
>>>> files in directory:
>>>> /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef
>>>> 14/06/17 18:05:08 DEBUG orm.CompilationManager: Got classfile:
>>>> /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.class
>>>> -> action.class
>>>> 14/06/17 18:05:08 DEBUG orm.CompilationManager: Finished writing jar
>>>> file /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.jar
>>>> 14/06/17 18:05:08 INFO teradata.TeradataManager: Beginning Teradata
>>>> table based import
>>>>
>>>> 1. I could successfully run "sqoop eval" to dump the table without
>>>> problem.
>>>>
>>>> What's wrong with my method?
>>>>
>>>
>>>
>>
>
Re: Sqoop hangs when import from Teradata
Posted by Benyi Wang <be...@gmail.com>.
I tried --verbose before, and no any affect. I waited for it until the
error message is shown.
On Wed, Jun 18, 2014 at 2:01 PM, Abraham Elmahrek <ab...@cloudera.com> wrote:
> Hey there,
>
> Glad every thing is working... and sorry it took so long. Did you add the
> --verbose flag to the beginning of the import statement? Or did you need to
> wait for the job to finish?
>
> -Abe
>
>
> On Wed, Jun 18, 2014 at 1:56 PM, Benyi Wang <be...@gmail.com> wrote:
>
>> It turns out that I have to provide "--split-by actionId". It took 53mins
>> to print out the exception in my system.
>>
>> 14/06/18 12:04:33 INFO teradata.TeradataManagerFactory: Loaded connector
>> factory for 'Cloudera Connector Powered by Teradata' on version 1.2c5
>> 14/06/18 12:04:33 INFO manager.SqlManager: Using default fetchSize of 1000
>> 14/06/18 12:04:33 INFO options.ExtraOptions: Parsing extra arguments
>> 14/06/18 12:04:33 INFO options.ExtraOptions: Loaded extra option for
>> batch size 1000
>> 14/06/18 12:04:33 INFO options.OptionsCompatibility: Checking options
>> compatibility
>> 14/06/18 12:04:35 INFO tool.CodeGenTool: Beginning code generation
>> 14/06/18 12:04:35 INFO teradata.TeradataManager: Converting table import
>> to query: SELECT * FROM "action"
>> 14/06/18 12:04:36 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
>> /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
>> Note:
>> /tmp/sqoop-bwang/compile/405654670de634debca58668abd0d1f4/action.java uses
>> or overrides a deprecated API.
>> Note: Recompile with -Xlint:deprecation for details.
>> 14/06/18 12:04:40 INFO orm.CompilationManager: Writing jar file:
>> /tmp/sqoop-bwang/compile/405654670de634debca58668abd0d1f4/action.jar
>> 14/06/18 12:04:40 INFO teradata.TeradataManager: Beginning Teradata table
>> based import
>> 14/06/18 12:57:25 ERROR sqoop.Sqoop: Got exception running Sqoop:
>> java.lang.RuntimeException: Table action does not have primary key. Please
>> use --split-by argument if applicable
>> java.lang.RuntimeException: Table action does not have primary key.
>> Please use --split-by argument if applicable
>> at
>> com.cloudera.connector.teradata.TeradataManager.getPrimaryKey(TeradataManager.java:175)
>> at
>> org.apache.sqoop.manager.SqlManager.getSplitColumn(SqlManager.java:546)
>> at
>> com.cloudera.connector.teradata.TeradataManager.importTable(TeradataManager.java:273)
>> at
>> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
>> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:506)
>> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>>
>>
>>
>> On Tue, Jun 17, 2014 at 6:09 PM, Benyi Wang <be...@gmail.com>
>> wrote:
>>
>>> I have trouble import Teradata table into HDFS:
>>>
>>> 1. Installed CDH using cloudera manager on CentOS 6.
>>> 2. Installed Sqoop-Teradata-Connector 1.2.5c5 using parcel.
>>> 3. Install Sqoop Client service, and deploy the configuration file.
>>> 4. Downloaded teradata jdbc driver v15.0.0.15 and v13.10.00.36, and
>>> extracted the jars in ~/apps/terajdbc.
>>> 5. export
>>> HADOOP_CLASSPATH=~/apps/terajdbc/terajdbc4.jar:~/apps/terajdbc/tdgssconfig.jar
>>> 6. sqoop import --connect jdbc:teradata://teraserver/DATABASE=MYDB
>>> --username bewang -P --table action --num-mappers 1, then it just hangs
>>> after creating jar. The table 'action' is pretty small, only 161 rows.
>>>
>>> 14/06/17 18:05:08 INFO orm.CompilationManager: Writing jar file:
>>> /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.jar
>>> 14/06/17 18:05:08 DEBUG orm.CompilationManager: Scanning for .class
>>> files in directory:
>>> /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef
>>> 14/06/17 18:05:08 DEBUG orm.CompilationManager: Got classfile:
>>> /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.class
>>> -> action.class
>>> 14/06/17 18:05:08 DEBUG orm.CompilationManager: Finished writing jar
>>> file /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.jar
>>> 14/06/17 18:05:08 INFO teradata.TeradataManager: Beginning Teradata
>>> table based import
>>>
>>> 1. I could successfully run "sqoop eval" to dump the table without
>>> problem.
>>>
>>> What's wrong with my method?
>>>
>>
>>
>
Re: Sqoop hangs when import from Teradata
Posted by Abraham Elmahrek <ab...@cloudera.com>.
Hey there,
Glad every thing is working... and sorry it took so long. Did you add the
--verbose flag to the beginning of the import statement? Or did you need to
wait for the job to finish?
-Abe
On Wed, Jun 18, 2014 at 1:56 PM, Benyi Wang <be...@gmail.com> wrote:
> It turns out that I have to provide "--split-by actionId". It took 53mins
> to print out the exception in my system.
>
> 14/06/18 12:04:33 INFO teradata.TeradataManagerFactory: Loaded connector
> factory for 'Cloudera Connector Powered by Teradata' on version 1.2c5
> 14/06/18 12:04:33 INFO manager.SqlManager: Using default fetchSize of 1000
> 14/06/18 12:04:33 INFO options.ExtraOptions: Parsing extra arguments
> 14/06/18 12:04:33 INFO options.ExtraOptions: Loaded extra option for batch
> size 1000
> 14/06/18 12:04:33 INFO options.OptionsCompatibility: Checking options
> compatibility
> 14/06/18 12:04:35 INFO tool.CodeGenTool: Beginning code generation
> 14/06/18 12:04:35 INFO teradata.TeradataManager: Converting table import
> to query: SELECT * FROM "action"
> 14/06/18 12:04:36 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
> /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
> Note:
> /tmp/sqoop-bwang/compile/405654670de634debca58668abd0d1f4/action.java uses
> or overrides a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 14/06/18 12:04:40 INFO orm.CompilationManager: Writing jar file:
> /tmp/sqoop-bwang/compile/405654670de634debca58668abd0d1f4/action.jar
> 14/06/18 12:04:40 INFO teradata.TeradataManager: Beginning Teradata table
> based import
> 14/06/18 12:57:25 ERROR sqoop.Sqoop: Got exception running Sqoop:
> java.lang.RuntimeException: Table action does not have primary key. Please
> use --split-by argument if applicable
> java.lang.RuntimeException: Table action does not have primary key. Please
> use --split-by argument if applicable
> at
> com.cloudera.connector.teradata.TeradataManager.getPrimaryKey(TeradataManager.java:175)
> at
> org.apache.sqoop.manager.SqlManager.getSplitColumn(SqlManager.java:546)
> at
> com.cloudera.connector.teradata.TeradataManager.importTable(TeradataManager.java:273)
> at
> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:506)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>
>
>
> On Tue, Jun 17, 2014 at 6:09 PM, Benyi Wang <be...@gmail.com> wrote:
>
>> I have trouble import Teradata table into HDFS:
>>
>> 1. Installed CDH using cloudera manager on CentOS 6.
>> 2. Installed Sqoop-Teradata-Connector 1.2.5c5 using parcel.
>> 3. Install Sqoop Client service, and deploy the configuration file.
>> 4. Downloaded teradata jdbc driver v15.0.0.15 and v13.10.00.36, and
>> extracted the jars in ~/apps/terajdbc.
>> 5. export
>> HADOOP_CLASSPATH=~/apps/terajdbc/terajdbc4.jar:~/apps/terajdbc/tdgssconfig.jar
>> 6. sqoop import --connect jdbc:teradata://teraserver/DATABASE=MYDB
>> --username bewang -P --table action --num-mappers 1, then it just hangs
>> after creating jar. The table 'action' is pretty small, only 161 rows.
>>
>> 14/06/17 18:05:08 INFO orm.CompilationManager: Writing jar file:
>> /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.jar
>> 14/06/17 18:05:08 DEBUG orm.CompilationManager: Scanning for .class files
>> in directory: /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef
>> 14/06/17 18:05:08 DEBUG orm.CompilationManager: Got classfile:
>> /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.class
>> -> action.class
>> 14/06/17 18:05:08 DEBUG orm.CompilationManager: Finished writing jar file
>> /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.jar
>> 14/06/17 18:05:08 INFO teradata.TeradataManager: Beginning Teradata table
>> based import
>>
>> 1. I could successfully run "sqoop eval" to dump the table without
>> problem.
>>
>> What's wrong with my method?
>>
>
>
Re: Sqoop hangs when import from Teradata
Posted by Benyi Wang <be...@gmail.com>.
It turns out that I have to provide "--split-by actionId". It took 53mins
to print out the exception in my system.
14/06/18 12:04:33 INFO teradata.TeradataManagerFactory: Loaded connector
factory for 'Cloudera Connector Powered by Teradata' on version 1.2c5
14/06/18 12:04:33 INFO manager.SqlManager: Using default fetchSize of 1000
14/06/18 12:04:33 INFO options.ExtraOptions: Parsing extra arguments
14/06/18 12:04:33 INFO options.ExtraOptions: Loaded extra option for batch
size 1000
14/06/18 12:04:33 INFO options.OptionsCompatibility: Checking options
compatibility
14/06/18 12:04:35 INFO tool.CodeGenTool: Beginning code generation
14/06/18 12:04:35 INFO teradata.TeradataManager: Converting table import to
query: SELECT * FROM "action"
14/06/18 12:04:36 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
Note: /tmp/sqoop-bwang/compile/405654670de634debca58668abd0d1f4/action.java
uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
14/06/18 12:04:40 INFO orm.CompilationManager: Writing jar file:
/tmp/sqoop-bwang/compile/405654670de634debca58668abd0d1f4/action.jar
14/06/18 12:04:40 INFO teradata.TeradataManager: Beginning Teradata table
based import
14/06/18 12:57:25 ERROR sqoop.Sqoop: Got exception running Sqoop:
java.lang.RuntimeException: Table action does not have primary key. Please
use --split-by argument if applicable
java.lang.RuntimeException: Table action does not have primary key. Please
use --split-by argument if applicable
at
com.cloudera.connector.teradata.TeradataManager.getPrimaryKey(TeradataManager.java:175)
at
org.apache.sqoop.manager.SqlManager.getSplitColumn(SqlManager.java:546)
at
com.cloudera.connector.teradata.TeradataManager.importTable(TeradataManager.java:273)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:506)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
On Tue, Jun 17, 2014 at 6:09 PM, Benyi Wang <be...@gmail.com> wrote:
> I have trouble import Teradata table into HDFS:
>
> 1. Installed CDH using cloudera manager on CentOS 6.
> 2. Installed Sqoop-Teradata-Connector 1.2.5c5 using parcel.
> 3. Install Sqoop Client service, and deploy the configuration file.
> 4. Downloaded teradata jdbc driver v15.0.0.15 and v13.10.00.36, and
> extracted the jars in ~/apps/terajdbc.
> 5. export
> HADOOP_CLASSPATH=~/apps/terajdbc/terajdbc4.jar:~/apps/terajdbc/tdgssconfig.jar
> 6. sqoop import --connect jdbc:teradata://teraserver/DATABASE=MYDB
> --username bewang -P --table action --num-mappers 1, then it just hangs
> after creating jar. The table 'action' is pretty small, only 161 rows.
>
> 14/06/17 18:05:08 INFO orm.CompilationManager: Writing jar file:
> /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.jar
> 14/06/17 18:05:08 DEBUG orm.CompilationManager: Scanning for .class files
> in directory: /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef
> 14/06/17 18:05:08 DEBUG orm.CompilationManager: Got classfile:
> /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.class
> -> action.class
> 14/06/17 18:05:08 DEBUG orm.CompilationManager: Finished writing jar file
> /tmp/sqoop-cloudera/compile/6c32f78ff095211b76e44a53cb3cdaef/action.jar
> 14/06/17 18:05:08 INFO teradata.TeradataManager: Beginning Teradata table
> based import
>
> 1. I could successfully run "sqoop eval" to dump the table without
> problem.
>
> What's wrong with my method?
>