You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Adarsh Sharma <ed...@gmail.com> on 2012/08/23 17:17:10 UTC
BigInt Support of postgresql in Sqoop
Hi all,
Please forgive if i violate any rule before posting in this mailing list .
I am using for some testing in my hadoop standalone set up.
Hadoop Version: 0.20.2-cdh3u5, 580d1d26c7ad6a7c6ba72950d8605e2c6fbc96cc
Sqoop Version : Sqoop 1.4.1-incubating
Also tried : Sqoop 1.4.0-incubating
Postgresql Versio : edb-psql (9.0.4.14)
I am able to export data from HDFS to postgresql but when I am trying to
import data from DB to hdfs , below problem arises :
hadoop@test123:~/project/
sqoop-1.4.1-incubating__hadoop-0.20$ bin/sqoop import --connect
jdbc:postgresql://localhost/hadooppipeline --table test_table --username
postgres --password postgres
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
12/08/23 19:25:19 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead.
12/08/23 19:25:19 INFO manager.SqlManager: Using default fetchSize of 1000
12/08/23 19:25:19 INFO tool.CodeGenTool: Beginning code generation
12/08/23 19:25:19 INFO manager.SqlManager: Executing SQL statement: SELECT
t.* FROM "test_table" AS t LIMIT 1
12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
12/08/23 19:25:19 ERROR orm.ClassWriter: No Java type for SQL type 2003 for
column rc_list
There are 4 bigint columns in the table. Please guide me if Sqoop support
for bigint columns or not.
I do some Rn D and find only one link but not able to solve the issue :
https://issues.cloudera.org/browse/SQOOP-48?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
Thanks
Re: BigInt Support of postgresql in Sqoop
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
You might consider utilizing Postgresql's function array_to_string to "join" the array into one string. You would have to change the import from --table to --query then thought.
Jarcec
On Fri, Aug 24, 2012 at 10:40:46AM +0530, Adarsh Sharma wrote:
> Thanks Jarcec for the update, So Sqoop is not suitable for shifting data
> from DB to HDFS , if some columns have integer[] or bigint[] datatypes.
>
> Is there any way i can sh*ift date having bigint[] datatypes from* postgresql
> DB to HDFS using Sqoop or I need to test another tool like Talend etc.
>
>
> Thanks
>
>
> On Thu, Aug 23, 2012 at 11:45 PM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
>
> > Hi Adarsh,
> > as far as I know then Sqoop should not have any issues with bigint data
> > type.
> >
> > Based on provided log fragment, It seems that you're having issues with
> > SQL type 2003 that should be ARRAY (see 1). I'm afraid that array is really
> > not supported in Sqoop at the moment.
> >
> > Jarcec
> >
> > 1:
> > http://docs.oracle.com/javase/1.5.0/docs/api/constant-values.html#java.sql.Types.ARRAY
> >
> > On Thu, Aug 23, 2012 at 08:47:10PM +0530, Adarsh Sharma wrote:
> > > Hi all,
> > >
> > > Please forgive if i violate any rule before posting in this mailing list
> > .
> > > I am using for some testing in my hadoop standalone set up.
> > >
> > > Hadoop Version: 0.20.2-cdh3u5, 580d1d26c7ad6a7c6ba72950d8605e2c6fbc96cc
> > > Sqoop Version : Sqoop 1.4.1-incubating
> > > Also tried : Sqoop 1.4.0-incubating
> > > Postgresql Versio : edb-psql (9.0.4.14)
> > >
> > >
> > > I am able to export data from HDFS to postgresql but when I am trying to
> > > import data from DB to hdfs , below problem arises :
> > > hadoop@test123:~/project/
> > > sqoop-1.4.1-incubating__hadoop-0.20$ bin/sqoop import --connect
> > > jdbc:postgresql://localhost/hadooppipeline --table test_table --username
> > > postgres --password postgres
> > > Warning: /usr/lib/hbase does not exist! HBase imports will fail.
> > > Please set $HBASE_HOME to the root of your HBase installation.
> > > 12/08/23 19:25:19 WARN tool.BaseSqoopTool: Setting your password on the
> > > command-line is insecure. Consider using -P instead.
> > > 12/08/23 19:25:19 INFO manager.SqlManager: Using default fetchSize of
> > 1000
> > > 12/08/23 19:25:19 INFO tool.CodeGenTool: Beginning code generation
> > > 12/08/23 19:25:19 INFO manager.SqlManager: Executing SQL statement:
> > SELECT
> > > t.* FROM "test_table" AS t LIMIT 1
> > > 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> > > 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> > > 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> > > 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> > > 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> > > 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> > > 12/08/23 19:25:19 ERROR orm.ClassWriter: No Java type for SQL type 2003
> > for
> > > column rc_list
> > >
> > > There are 4 bigint columns in the table. Please guide me if Sqoop support
> > > for bigint columns or not.
> > >
> > > I do some Rn D and find only one link but not able to solve the issue :
> > >
> > https://issues.cloudera.org/browse/SQOOP-48?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
> > >
> > >
> > > Thanks
> >
Re: BigInt Support of postgresql in Sqoop
Posted by Adarsh Sharma <ed...@gmail.com>.
Thanks Jarcec for the update, So Sqoop is not suitable for shifting data
from DB to HDFS , if some columns have integer[] or bigint[] datatypes.
Is there any way i can sh*ift date having bigint[] datatypes from* postgresql
DB to HDFS using Sqoop or I need to test another tool like Talend etc.
Thanks
On Thu, Aug 23, 2012 at 11:45 PM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
> Hi Adarsh,
> as far as I know then Sqoop should not have any issues with bigint data
> type.
>
> Based on provided log fragment, It seems that you're having issues with
> SQL type 2003 that should be ARRAY (see 1). I'm afraid that array is really
> not supported in Sqoop at the moment.
>
> Jarcec
>
> 1:
> http://docs.oracle.com/javase/1.5.0/docs/api/constant-values.html#java.sql.Types.ARRAY
>
> On Thu, Aug 23, 2012 at 08:47:10PM +0530, Adarsh Sharma wrote:
> > Hi all,
> >
> > Please forgive if i violate any rule before posting in this mailing list
> .
> > I am using for some testing in my hadoop standalone set up.
> >
> > Hadoop Version: 0.20.2-cdh3u5, 580d1d26c7ad6a7c6ba72950d8605e2c6fbc96cc
> > Sqoop Version : Sqoop 1.4.1-incubating
> > Also tried : Sqoop 1.4.0-incubating
> > Postgresql Versio : edb-psql (9.0.4.14)
> >
> >
> > I am able to export data from HDFS to postgresql but when I am trying to
> > import data from DB to hdfs , below problem arises :
> > hadoop@test123:~/project/
> > sqoop-1.4.1-incubating__hadoop-0.20$ bin/sqoop import --connect
> > jdbc:postgresql://localhost/hadooppipeline --table test_table --username
> > postgres --password postgres
> > Warning: /usr/lib/hbase does not exist! HBase imports will fail.
> > Please set $HBASE_HOME to the root of your HBase installation.
> > 12/08/23 19:25:19 WARN tool.BaseSqoopTool: Setting your password on the
> > command-line is insecure. Consider using -P instead.
> > 12/08/23 19:25:19 INFO manager.SqlManager: Using default fetchSize of
> 1000
> > 12/08/23 19:25:19 INFO tool.CodeGenTool: Beginning code generation
> > 12/08/23 19:25:19 INFO manager.SqlManager: Executing SQL statement:
> SELECT
> > t.* FROM "test_table" AS t LIMIT 1
> > 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> > 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> > 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> > 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> > 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> > 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> > 12/08/23 19:25:19 ERROR orm.ClassWriter: No Java type for SQL type 2003
> for
> > column rc_list
> >
> > There are 4 bigint columns in the table. Please guide me if Sqoop support
> > for bigint columns or not.
> >
> > I do some Rn D and find only one link but not able to solve the issue :
> >
> https://issues.cloudera.org/browse/SQOOP-48?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
> >
> >
> > Thanks
>
Re: BigInt Support of postgresql in Sqoop
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
Hi Adarsh,
as far as I know then Sqoop should not have any issues with bigint data type.
Based on provided log fragment, It seems that you're having issues with SQL type 2003 that should be ARRAY (see 1). I'm afraid that array is really not supported in Sqoop at the moment.
Jarcec
1: http://docs.oracle.com/javase/1.5.0/docs/api/constant-values.html#java.sql.Types.ARRAY
On Thu, Aug 23, 2012 at 08:47:10PM +0530, Adarsh Sharma wrote:
> Hi all,
>
> Please forgive if i violate any rule before posting in this mailing list .
> I am using for some testing in my hadoop standalone set up.
>
> Hadoop Version: 0.20.2-cdh3u5, 580d1d26c7ad6a7c6ba72950d8605e2c6fbc96cc
> Sqoop Version : Sqoop 1.4.1-incubating
> Also tried : Sqoop 1.4.0-incubating
> Postgresql Versio : edb-psql (9.0.4.14)
>
>
> I am able to export data from HDFS to postgresql but when I am trying to
> import data from DB to hdfs , below problem arises :
> hadoop@test123:~/project/
> sqoop-1.4.1-incubating__hadoop-0.20$ bin/sqoop import --connect
> jdbc:postgresql://localhost/hadooppipeline --table test_table --username
> postgres --password postgres
> Warning: /usr/lib/hbase does not exist! HBase imports will fail.
> Please set $HBASE_HOME to the root of your HBase installation.
> 12/08/23 19:25:19 WARN tool.BaseSqoopTool: Setting your password on the
> command-line is insecure. Consider using -P instead.
> 12/08/23 19:25:19 INFO manager.SqlManager: Using default fetchSize of 1000
> 12/08/23 19:25:19 INFO tool.CodeGenTool: Beginning code generation
> 12/08/23 19:25:19 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM "test_table" AS t LIMIT 1
> 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> 12/08/23 19:25:19 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
> 12/08/23 19:25:19 ERROR orm.ClassWriter: No Java type for SQL type 2003 for
> column rc_list
>
> There are 4 bigint columns in the table. Please guide me if Sqoop support
> for bigint columns or not.
>
> I do some Rn D and find only one link but not able to solve the issue :
> https://issues.cloudera.org/browse/SQOOP-48?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
>
>
> Thanks