Re: Github Sqoop

[Jarek Jarcec Cecho <ja...@apache.org>]

Hi Pablo,
I’m adding  Sqoop user mailing list user@sqoop.apache.org - please sing up to this mailing list as instructed in [1].

If you use s3:/ as the underlaying filesystem, Sqoop uses Hadoop’s S3 implementation. I honestly do not know if the S3 implementation uses secured (~SSL) connection underneath even though I would expect that. Looking around internet it seems that these days people are suggesting to use s3a:// instead of s3:// [2].

Jarcec

Links:
1: http://sqoop.apache.org/mail-lists.html
2: https://wiki.apache.org/hadoop/AmazonS3

> On Sep 16, 2015, at 2:50 AM, Pablo Torre <pa...@gmail.com> wrote:
> 
> Hi Jarek,
> 
> I need your help.
> 
> I would like to ask you a question and see if you can help me. I have this conf file:
> 
> Hi,
> 
> I have this config file "sqoop.conf":
> 
> #
> # Options file for Sqoop
> #
> # Specifies the tool being invoked
> import
> # Connect parameter and value
> --connect
> jdbc:sqlserver://xxx.xxx.x.xx\SQLEXPRESS;database=TEST
> # Username parameter and value
> --username
> xxxx
> --password
> xxxx
> --verbose
> --target-dir
> s3://......
> --split-by
> age
> --table
> dbo.users
> --fields-terminated-by
> ,
> --input-fields-terminated-by
> ,
> --check-column
> updated_at
> --incremental
> lastmodified
> --last-value
> 2015-02-07 10:33:47
> 
> 
> It is working perfectly, I am using Sqoop to import data from a SQL Server (through a VPN) to Amazon S3. I now that this connection is safe, but what happen when Sqoop is moving this data to Amazon S3. Is it safe this connection? Is using Sqoop underneath a security connection? I don't know how it works.
> 
> Best,
> 
> 
> 
> 
> 2015-05-04 12:53 GMT+02:00 Pablo Torre <pa...@gmail.com>:
> Hi,
> 
> It works!! Thanks a lot for your help. 
> 
> 
> Best,
> Pablo
> 
> 2015-05-01 19:01 GMT+02:00 Jarek Jarcec Cecho <ja...@apache.org>:
> I believe that the problem is because you’re explicitly using the —driver argument Pablo. Could you drop it?
> 
> The —driver option is a bit deceptive in Sqoop and I generally recommend not to use it at all unless absolutely required. You can read why in the following blog post:
> 
> http://blog.cloudera.com/blog/2013/09/understanding-connectors-and-drivers-in-the-world-of-sqoop/
> 
> Jarcec
> 
> > On May 1, 2015, at 9:28 AM, Pablo Torre <pa...@gmail.com> wrote:
> >
> > Hi,
> >
> > I have this config file "sqoop.conf":
> >
> > #
> > # Options file for Sqoop
> > #
> > # Specifies the tool being invoked
> > import
> > # Connect parameter and value
> > --connect
> > jdbc:sqlserver://xxx.xxx.x.xx\SQLEXPRESS;database=TEST
> > # Username parameter and value
> > --username
> > xxxx
> > --password
> > xxxx
> > --driver
> > com.microsoft.sqlserver.jdbc.SQLServerDriver
> > --verbose
> > --target-dir
> > /user/pablotorrerodriguez/sqlserverusers5/
> > --split-by
> > age
> > --table
> > dbo.users
> > --fields-terminated-by
> > ,
> > --input-fields-terminated-by
> > ,
> > --check-column
> > updated_at
> > --incremental
> > lastmodified
> > --last-value
> > 2015-02-07 10:33:47
> >
> >
> > And I run this command line:
> >
> >  sqoop --options-file sqoop.conf
> >
> > But I get this error "com.microsoft.sqlserver.jdbc.SQLServerException: Incorrent sintaxis near to ')'." I am using hadoop 1.2.1 and sqoop 1.4.5. And in the folder sqoop-1.4.5/lib I have the driver sqljdbc4.jar for  Microsoft SQL Server.
> > The incremental imports with Microsoft SQL Server give me problems.
> >
> >
> > Best,
> >
> >
> > 2015-05-01 16:28 GMT+02:00 Jarek Jarcec Cecho <ja...@apache.org>:
> > I’m adding user@sqoop.apache.org mailing list as this seems to be normal user question Please do register in the mailing list Pablo, you can find instructions in [1].
> >
> > Could you share with us your entire Sqoop command line?
> >
> > Jarcec
> >
> > 1: http://sqoop.apache.org/mail-lists.html
> >
> > > On May 1, 2015, at 1:10 AM, Pablo Torre <pa...@gmail.com> wrote:
> > >
> > > Hi Jarek.
> > >
> > > I am taking a look at code in Sqoop 1.4.5 and in the java class Sqlmanager.java I can see that it is using "SELECT TIMESTAMP;" instead of use "SELECT TIMESTAMP()". But I still get the error "com.microsoft.sqlserver.jdbc.SQLServerException: Incorrent sintaxis near to ')'."
> > >
> > > I was testing it in Microsoft SQL Server Management and If I run the query "SELECT TIMESTAMP()" I get the same error.
> > >
> > > I don't know why sqoop is trying to execute the wrong query if the source code is right.
> > >
> > > Please help me with this.
> > >
> > >
> > >
> > > 2015-04-20 16:41 GMT+02:00 Jarek Jarcec Cecho <ja...@apache.org>:
> > > I believe that this is already fixed in Sqoop 1.4.5. I would suggest to sign up to Sqoop user mailing list (instructions are at [1]) and ask for help :)
> > >
> > > Jarcec
> > >
> > > Links:
> > > 1: http://sqoop.apache.org/mail-lists.html
> > >
> > > > On Apr 19, 2015, at 2:12 PM, Pablo Torre <pa...@gmail.com> wrote:
> > > >
> > > > Thanks Jarek,
> > > >
> > > > I was using Sqoop 1.4.5, maybe I have to install a new version or update the code.
> > > >
> > > > Best,
> > > > Pablo
> > > >
> > > > 2015-04-19 20:42 GMT+02:00 Jarek Jarcec Cecho <ja...@apache.org>:
> > > > Hi Pablo,
> > > > I’m always excited to see a new contributor on Sqoop project. Take a look at quick “How to contribute” wiki page that contains all the details about how to contribute to Sqoop:
> > > >
> > > > https://cwiki.apache.org/confluence/display/SQOOP/How+to+Contribute
> > > >
> > > > For your particular issue, I feel that it’s long solved as the Microsoft Sql connector is already using the correct query, check out the latest source code:
> > > >
> > > > https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/manager/SQLServerManager.java#L251
> > > >
> > > > Jarcec
> > > >
> > > > > On Apr 17, 2015, at 2:24 AM, Pablo Torre <pa...@gmail.com> wrote:
> > > > >
> > > > > Hi Jarcec,
> > > > >
> > > > > My name is Pablo. I am writing this email because I saw that you are working on the project sqoop on github. https://github.com/apache/sqoop
> > > > >
> > > > > There is a bug that I would like to share with you and maybe you can help me and say me how can I modified the source code or contribute to the project:
> > > > >
> > > > > I am trying to use incremental imports with sqoop using the option lastmodified, but with MS SQLSERVER fails. I found this article about it https://issues.cloudera.org/browse/SQOOP-229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> > > > >
> > > > > "I see in the source code for SqlManager.java that the query being used in getCurTimestampQuery() is "select CURRENT_TIMESTAMP()" this is not compatible with MSSQL - for MSSQL it would be just "select
> > > > > CURRENT_TIMESTAMP" (keyword not a function)."
> > > > >
> > > > > I would like to modify that class and compile the code to get it working. But I don't know how to do it.
> > > > >
> > > > > Do you know if there is any patch for it?
> > > > > If you can help me with this issue.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > My Github: https://github.com/sylvinho81
> > > > >
> > > > > --
> > > > > Pablo Torre.
> > > > > Freelance software engineer and Ruby on Rails developer.
> > > > > Oleiros (Coruña)
> > > > > Personal site
> > > > > My blog
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Pablo Torre.
> > > > Freelance software engineer and Ruby on Rails developer.
> > > > Oleiros (Coruña)
> > > > Personal site
> > > > My blog
> > >
> > >
> > >
> > >
> > > --
> > > Pablo Torre.
> > > Freelance software engineer and Ruby on Rails developer.
> > > Oleiros (Coruña)
> > > Personal site
> > > My blog
> >
> >
> >
> >
> > --
> > Pablo Torre.
> > Freelance software engineer and Ruby on Rails developer.
> > Oleiros (Coruña)
> > Personal site
> > My blog
> 
> 
> 
> 
> -- 
> Pablo Torre.
> Freelance software engineer and Ruby on Rails developer.
> Oleiros (Coruña)
> Personal site
> My blog
> 
> 
> 
> -- 
> Pablo Torre.
> Freelance software engineer and Ruby on Rails developer.
> Oleiros (Coruña)
> Personal site
> My blog