You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Lei Lei <le...@gmail.com> on 2014/04/08 15:34:53 UTC
One question about "sqoop export"
Hi all,
I have one question about "sqoop export". Thanks!
My Hadoop version is 1.2.1, and it works well; my sqoop version is
1.4.4.
I also installed MySQL whose version is 5.1.73. In MySQL, I created one
table as below:
create table dimension_result (
result int not null primary key,
description VARCHAR(32) not null
);
On HDFS, I have a file called dimension_result.txt under /tmp/dr
See below.
hadoop fs -cat /tmp/dr/dimension_result.txt
1,BlackWin
2,WhiteWin
0,Draw
-1,Unknown
I want to export the content of dimension_result.txt to the MySQL
table, but it always failed.
My command is:
sqoop-export --connect jdbc:mysql://127.0.0.1:3306/test --driver
com.mysql.jdbc.Driver --username hadoop --table dimension_result
--export-dir /tmp/dr
I also tried:
sqoop-export --connect jdbc:mysql://127.0.0.1:3306/test --driver
com.mysql.jdbc.Driver --username hadoop --table dimension_result
--export-dir /tmp/dr/dimension_result.txt
(My MySQL instance does not need password, and the following command
can succeed.
sqoop-list-databases --connect jdbc:mysql://127.0.0.1:3306 --username
hadoop )
The error info is as below. Does anyone know why? Thanks!
14/04/08 21:33:03 INFO mapred.JobClient: Task Id :
attempt_201403071512_0021_m_000000_1, Status : FAILED
java.io.IOException:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
link failure
The last packet sent successfully to the server was 0 milliseconds ago. The
driver has not received any packets from the server.
at
org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:627)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The
driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358)
at
com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2498)
at
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2535)
at
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at
org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:294)
at
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.<init>(AsyncSqlRecordWriter.java:76)
at
org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.<init>(ExportOutputFormat.java:95)
at
org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:77)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:241)
at
com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:308)
... 26 more
Regards,
Finix
Re: One question about "sqoop export"
Posted by Lei Lei <le...@gmail.com>.
Ah, I think you are right, Jarek. I will try it later. Thanks.
2014-04-08 22:02 GMT+08:00 Jarek Jarcec Cecho <ja...@apache.org>:
> On second look - you are specifying "localhost" in the connection URL. As
> Sqoop will open connections from different machines, this won't work. You
> always need to use fully qualified domain name.
>
> Jarcec
>
> On Tue, Apr 08, 2014 at 06:57:01AM -0700, Jarek Jarcec Cecho wrote:
> > Hi Lei,
> > please do include entire log generated with parameter --verbose next
> time. The same (well similar) exception can be thrown from multiple places
> and hence having entire log is usually critical to triage the problem.
> >
> > In your case it seems that Sqoop is able to get all the way down to
> mapreduce job submission and hence I'm wondering if *all* your Hadoop nodes
> can access the database?
> >
> > Jarcec
> >
> > On Tue, Apr 08, 2014 at 09:34:53PM +0800, Lei Lei wrote:
> > > Hi all,
> > > I have one question about "sqoop export". Thanks!
> > > My Hadoop version is 1.2.1, and it works well; my sqoop version is
> > > 1.4.4.
> > > I also installed MySQL whose version is 5.1.73. In MySQL, I
> created one
> > > table as below:
> > >
> > > create table dimension_result (
> > > result int not null primary key,
> > > description VARCHAR(32) not null
> > > );
> > >
> > > On HDFS, I have a file called dimension_result.txt under /tmp/dr
> > > See below.
> > > hadoop fs -cat /tmp/dr/dimension_result.txt
> > > 1,BlackWin
> > > 2,WhiteWin
> > > 0,Draw
> > > -1,Unknown
> > >
> > > I want to export the content of dimension_result.txt to the MySQL
> > > table, but it always failed.
> > > My command is:
> > > sqoop-export --connect jdbc:mysql://127.0.0.1:3306/test --driver
> > > com.mysql.jdbc.Driver --username hadoop --table dimension_result
> > > --export-dir /tmp/dr
> > >
> > > I also tried:
> > > sqoop-export --connect jdbc:mysql://127.0.0.1:3306/test --driver
> > > com.mysql.jdbc.Driver --username hadoop --table dimension_result
> > > --export-dir /tmp/dr/dimension_result.txt
> > >
> > > (My MySQL instance does not need password, and the following
> command
> > > can succeed.
> > > sqoop-list-databases --connect jdbc:mysql://127.0.0.1:3306 --username
> > > hadoop )
> > >
> > > The error info is as below. Does anyone know why? Thanks!
> > >
> > > 14/04/08 21:33:03 INFO mapred.JobClient: Task Id :
> > > attempt_201403071512_0021_m_000000_1, Status : FAILED
> > > java.io.IOException:
> > > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> > > link failure
> > >
> > > The last packet sent successfully to the server was 0 milliseconds
> ago. The
> > > driver has not received any packets from the server.
> > > at
> > >
> org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79)
> > > at
> > >
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:627)
> > > at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
> > > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at javax.security.auth.Subject.doAs(Subject.java:415)
> > > at
> > >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:249)
> > > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> > > Communications link failure
> > >
> > > The last packet sent successfully to the server was 0 milliseconds
> ago. The
> > > driver has not received any packets from the server.
> > > at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > Method)
> > > at
> > >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> > > at
> > >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > > at
> java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> > > at
> > >
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
> > > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358)
> > > at
> > > com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2498)
> > > at
> > >
> com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2535)
> > > at
> > > com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
> > > at
> com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
> > > at
> com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
> > > at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > Method)
> > > at
> > >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> > > at
> > >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > > at
> java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> > > at
> > > com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
> > > at
> > >
> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
> > > at java.sql.DriverManager.getConnection(DriverManager.java:571)
> > > at java.sql.DriverManager.getConnection(DriverManager.java:215)
> > > at
> > >
> org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:294)
> > > at
> > >
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.<init>(AsyncSqlRecordWriter.java:76)
> > > at
> > >
> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.<init>(ExportOutputFormat.java:95)
> > > at
> > >
> org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:77)
> > > ... 8 more
> > > Caused by: java.net.ConnectException: Connection refused
> > > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > at
> > >
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> > > at
> > >
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> > > at
> > >
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> > > at java.net.Socket.connect(Socket.java:579)
> > > at java.net.Socket.connect(Socket.java:528)
> > > at java.net.Socket.<init>(Socket.java:425)
> > > at java.net.Socket.<init>(Socket.java:241)
> > > at
> > >
> com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
> > > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:308)
> > > ... 26 more
> > >
> > >
> > > Regards,
> > > Finix
>
>
>
Re: One question about "sqoop export"
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
On second look - you are specifying "localhost" in the connection URL. As Sqoop will open connections from different machines, this won't work. You always need to use fully qualified domain name.
Jarcec
On Tue, Apr 08, 2014 at 06:57:01AM -0700, Jarek Jarcec Cecho wrote:
> Hi Lei,
> please do include entire log generated with parameter --verbose next time. The same (well similar) exception can be thrown from multiple places and hence having entire log is usually critical to triage the problem.
>
> In your case it seems that Sqoop is able to get all the way down to mapreduce job submission and hence I'm wondering if *all* your Hadoop nodes can access the database?
>
> Jarcec
>
> On Tue, Apr 08, 2014 at 09:34:53PM +0800, Lei Lei wrote:
> > Hi all,
> > I have one question about "sqoop export". Thanks!
> > My Hadoop version is 1.2.1, and it works well; my sqoop version is
> > 1.4.4.
> > I also installed MySQL whose version is 5.1.73. In MySQL, I created one
> > table as below:
> >
> > create table dimension_result (
> > result int not null primary key,
> > description VARCHAR(32) not null
> > );
> >
> > On HDFS, I have a file called dimension_result.txt under /tmp/dr
> > See below.
> > hadoop fs -cat /tmp/dr/dimension_result.txt
> > 1,BlackWin
> > 2,WhiteWin
> > 0,Draw
> > -1,Unknown
> >
> > I want to export the content of dimension_result.txt to the MySQL
> > table, but it always failed.
> > My command is:
> > sqoop-export --connect jdbc:mysql://127.0.0.1:3306/test --driver
> > com.mysql.jdbc.Driver --username hadoop --table dimension_result
> > --export-dir /tmp/dr
> >
> > I also tried:
> > sqoop-export --connect jdbc:mysql://127.0.0.1:3306/test --driver
> > com.mysql.jdbc.Driver --username hadoop --table dimension_result
> > --export-dir /tmp/dr/dimension_result.txt
> >
> > (My MySQL instance does not need password, and the following command
> > can succeed.
> > sqoop-list-databases --connect jdbc:mysql://127.0.0.1:3306 --username
> > hadoop )
> >
> > The error info is as below. Does anyone know why? Thanks!
> >
> > 14/04/08 21:33:03 INFO mapred.JobClient: Task Id :
> > attempt_201403071512_0021_m_000000_1, Status : FAILED
> > java.io.IOException:
> > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> > link failure
> >
> > The last packet sent successfully to the server was 0 milliseconds ago. The
> > driver has not received any packets from the server.
> > at
> > org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79)
> > at
> > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:627)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> > at org.apache.hadoop.mapred.Child.main(Child.java:249)
> > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> > Communications link failure
> >
> > The last packet sent successfully to the server was 0 milliseconds ago. The
> > driver has not received any packets from the server.
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> > at
> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> > at
> > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
> > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358)
> > at
> > com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2498)
> > at
> > com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2535)
> > at
> > com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
> > at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
> > at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> > at
> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> > at
> > com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
> > at
> > com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
> > at java.sql.DriverManager.getConnection(DriverManager.java:571)
> > at java.sql.DriverManager.getConnection(DriverManager.java:215)
> > at
> > org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:294)
> > at
> > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.<init>(AsyncSqlRecordWriter.java:76)
> > at
> > org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.<init>(ExportOutputFormat.java:95)
> > at
> > org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:77)
> > ... 8 more
> > Caused by: java.net.ConnectException: Connection refused
> > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > at
> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> > at
> > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> > at
> > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> > at java.net.Socket.connect(Socket.java:579)
> > at java.net.Socket.connect(Socket.java:528)
> > at java.net.Socket.<init>(Socket.java:425)
> > at java.net.Socket.<init>(Socket.java:241)
> > at
> > com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
> > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:308)
> > ... 26 more
> >
> >
> > Regards,
> > Finix
Re: One question about "sqoop export"
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
Hi Lei,
please do include entire log generated with parameter --verbose next time. The same (well similar) exception can be thrown from multiple places and hence having entire log is usually critical to triage the problem.
In your case it seems that Sqoop is able to get all the way down to mapreduce job submission and hence I'm wondering if *all* your Hadoop nodes can access the database?
Jarcec
On Tue, Apr 08, 2014 at 09:34:53PM +0800, Lei Lei wrote:
> Hi all,
> I have one question about "sqoop export". Thanks!
> My Hadoop version is 1.2.1, and it works well; my sqoop version is
> 1.4.4.
> I also installed MySQL whose version is 5.1.73. In MySQL, I created one
> table as below:
>
> create table dimension_result (
> result int not null primary key,
> description VARCHAR(32) not null
> );
>
> On HDFS, I have a file called dimension_result.txt under /tmp/dr
> See below.
> hadoop fs -cat /tmp/dr/dimension_result.txt
> 1,BlackWin
> 2,WhiteWin
> 0,Draw
> -1,Unknown
>
> I want to export the content of dimension_result.txt to the MySQL
> table, but it always failed.
> My command is:
> sqoop-export --connect jdbc:mysql://127.0.0.1:3306/test --driver
> com.mysql.jdbc.Driver --username hadoop --table dimension_result
> --export-dir /tmp/dr
>
> I also tried:
> sqoop-export --connect jdbc:mysql://127.0.0.1:3306/test --driver
> com.mysql.jdbc.Driver --username hadoop --table dimension_result
> --export-dir /tmp/dr/dimension_result.txt
>
> (My MySQL instance does not need password, and the following command
> can succeed.
> sqoop-list-databases --connect jdbc:mysql://127.0.0.1:3306 --username
> hadoop )
>
> The error info is as below. Does anyone know why? Thanks!
>
> 14/04/08 21:33:03 INFO mapred.JobClient: Task Id :
> attempt_201403071512_0021_m_000000_1, Status : FAILED
> java.io.IOException:
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link failure
>
> The last packet sent successfully to the server was 0 milliseconds ago. The
> driver has not received any packets from the server.
> at
> org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:627)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
>
> The last packet sent successfully to the server was 0 milliseconds ago. The
> driver has not received any packets from the server.
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
> at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358)
> at
> com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2498)
> at
> com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2535)
> at
> com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
> at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
> at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at
> com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
> at
> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
> at java.sql.DriverManager.getConnection(DriverManager.java:571)
> at java.sql.DriverManager.getConnection(DriverManager.java:215)
> at
> org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:294)
> at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.<init>(AsyncSqlRecordWriter.java:76)
> at
> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.<init>(ExportOutputFormat.java:95)
> at
> org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:77)
> ... 8 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at java.net.Socket.connect(Socket.java:528)
> at java.net.Socket.<init>(Socket.java:425)
> at java.net.Socket.<init>(Socket.java:241)
> at
> com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
> at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:308)
> ... 26 more
>
>
> Regards,
> Finix
Re: One question about "sqoop export"
Posted by Abraham Elmahrek <ab...@cloudera.com>.
Try removing the "--driver". Sqoop should automatically choose the correct
driver/manager based on your JDBC connection string.
On Tue, Apr 8, 2014 at 7:52 AM, Sandipan.Ghosh <Sa...@target.com>wrote:
> Do you have write permission to MySql?
>
>
>
> *From:* Lei Lei [mailto:learnerfinix@gmail.com]
> *Sent:* Tuesday, April 08, 2014 7:05 PM
> *To:* user@sqoop.apache.org
> *Subject:* One question about "sqoop export"
>
>
>
> Hi all,
>
> I have one question about "sqoop export". Thanks!
>
> My Hadoop version is 1.2.1, and it works well; my sqoop version is
> 1.4.4.
>
> I also installed MySQL whose version is 5.1.73. In MySQL, I created
> one table as below:
>
>
>
> create table dimension_result (
>
> result int not null primary key,
>
> description VARCHAR(32) not null
>
> );
>
>
>
> On HDFS, I have a file called dimension_result.txt under /tmp/dr
>
> See below.
>
> hadoop fs -cat /tmp/dr/dimension_result.txt
>
> 1,BlackWin
>
> 2,WhiteWin
>
> 0,Draw
>
> -1,Unknown
>
>
>
> I want to export the content of dimension_result.txt to the MySQL
> table, but it always failed.
>
> My command is:
>
> sqoop-export --connect jdbc:mysql://127.0.0.1:3306/test --driver
> com.mysql.jdbc.Driver --username hadoop --table dimension_result
> --export-dir /tmp/dr
>
>
>
> I also tried:
>
> sqoop-export --connect jdbc:mysql://127.0.0.1:3306/test --driver
> com.mysql.jdbc.Driver --username hadoop --table dimension_result
> --export-dir /tmp/dr/dimension_result.txt
>
>
>
> (My MySQL instance does not need password, and the following command
> can succeed.
>
> sqoop-list-databases --connect jdbc:mysql://127.0.0.1:3306 --username
> hadoop )
>
>
>
> The error info is as below. Does anyone know why? Thanks!
>
>
>
> 14/04/08 21:33:03 INFO mapred.JobClient: Task Id :
> attempt_201403071512_0021_m_000000_1, Status : FAILED
>
> java.io.IOException:
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link failure
>
>
>
> The last packet sent successfully to the server was 0 milliseconds ago.
> The driver has not received any packets from the server.
>
> at
> org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79)
>
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:627)
>
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
>
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
>
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
>
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
>
>
>
> The last packet sent successfully to the server was 0 milliseconds ago.
> The driver has not received any packets from the server.
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
>
> at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358)
>
> at
> com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2498)
>
> at
> com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2535)
>
> at
> com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
>
> at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
>
> at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>
> at
> com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
>
> at
> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
>
> at java.sql.DriverManager.getConnection(DriverManager.java:571)
>
> at java.sql.DriverManager.getConnection(DriverManager.java:215)
>
> at
> org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:294)
>
> at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.<init>(AsyncSqlRecordWriter.java:76)
>
> at
> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.<init>(ExportOutputFormat.java:95)
>
> at
> org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:77)
>
> ... 8 more
>
> Caused by: java.net.ConnectException: Connection refused
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:579)
>
> at java.net.Socket.connect(Socket.java:528)
>
> at java.net.Socket.<init>(Socket.java:425)
>
> at java.net.Socket.<init>(Socket.java:241)
>
> at
> com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
>
> at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:308)
>
> ... 26 more
>
>
>
>
>
> Regards,
>
> Finix
>
>
>
RE: One question about "sqoop export"
Posted by "Sandipan.Ghosh" <Sa...@target.com>.
Do you have write permission to MySql?
From: Lei Lei [mailto:learnerfinix@gmail.com]
Sent: Tuesday, April 08, 2014 7:05 PM
To: user@sqoop.apache.org
Subject: One question about "sqoop export"
Hi all,
I have one question about "sqoop export". Thanks!
My Hadoop version is 1.2.1, and it works well; my sqoop version is 1.4.4.
I also installed MySQL whose version is 5.1.73. In MySQL, I created one table as below:
create table dimension_result (
result int not null primary key,
description VARCHAR(32) not null
);
On HDFS, I have a file called dimension_result.txt under /tmp/dr
See below.
hadoop fs -cat /tmp/dr/dimension_result.txt
1,BlackWin
2,WhiteWin
0,Draw
-1,Unknown
I want to export the content of dimension_result.txt to the MySQL table, but it always failed.
My command is:
sqoop-export --connect jdbc:mysql://127.0.0.1:3306/test<http://127.0.0.1:3306/test> --driver com.mysql.jdbc.Driver --username hadoop --table dimension_result --export-dir /tmp/dr
I also tried:
sqoop-export --connect jdbc:mysql://127.0.0.1:3306/test<http://127.0.0.1:3306/test> --driver com.mysql.jdbc.Driver --username hadoop --table dimension_result --export-dir /tmp/dr/dimension_result.txt
(My MySQL instance does not need password, and the following command can succeed.
sqoop-list-databases --connect jdbc:mysql://127.0.0.1:3306<http://127.0.0.1:3306> --username hadoop )
The error info is as below. Does anyone know why? Thanks!
14/04/08 21:33:03 INFO mapred.JobClient: Task Id : attempt_201403071512_0021_m_000000_1, Status : FAILED
java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:627)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2498)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2535)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:294)
at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.<init>(AsyncSqlRecordWriter.java:76)
at org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.<init>(ExportOutputFormat.java:95)
at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:77)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:241)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:308)
... 26 more
Regards,
Finix