You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Jonathan Kaufman <jk...@footlocker.com> on 2016/09/07 15:47:30 UTC
Help using Scoop 1.4.6 with Kerberos Authentication on Import with
MapReduce.
Hello all,
I apologize in advance if this seems like a silly question, however I've
not been able to determine if Scoop supports Kerberos delegation on
import, down to the MapReduce process.
I have a user who has a valid Kerberos ticket that supports forwarding,
additionally they are working on a completely kerberized Hadoop cluster
and all hosts in the cluster are trusted for kerberos delegation.
We have been able to use Scoop with Kerberos Authentication when
connecting to a Microsoft SQL Server using the following command:
sqoop
list-databases --connect
"jdbc:sqlserver://hostname.domain.com;integratedSecurity=true;authenticationScheme=JavaKerberos"
--username 'DOMAIN\USER'
So we tried to do an import using Kerberos, but it has failed with an
error that indicates that MapReduce does not have a kerberos ticket.
Here is the command used on the Sqoop import.
sqoop import --connect
"jdbc:sqlserver://FQDN;integratedSecurity=true;authenticationScheme=JavaKerberos;database=database_name"
--username 'Domain\user' --query 'select * from table where $CONDITIONS'
--split-by Abc_Number --target-dir /user/user/Sql
Here is the error:
INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service:
ha-hdfs:NNHA, Ident: (HDFS_DELEGATION_TOKEN token 2496 for ) INFO
impl.YarnClientImpl: Submitted application INFO mapreduce.Job: The url to
track the job: INFO mapreduce.Job: Running job: job_1 INFO mapreduce.Job:
Job job_1 running in uber mode : false INFO mapreduce.Job: map 0% reduce
0% INFO mapreduce.Job: Task Id : attempt_1470416754637_0238_m_000003_0,
Status : FAILED Error: java.lang.RuntimeException:
java.lang.RuntimeException:
com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication
failed. ClientConnectionId:366c38c7-c at
org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by:
java.lang.RuntimeException:
com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication
failed. ClientConnectionId:366c38c7-c at
org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at
org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more Caused by: com.microsoft.sqlserver.jdbc.SQLServerException:
Integrated authentication failed. ClientConnectionId:366c38c7-c at
com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:176)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication.GenerateClientContext(KerbAuthentication.java:296)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4084)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3159)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.access$100(SQLServerConnection.java:42)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3122)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505) at
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1980)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1627)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1458)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:772)
at
com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1168)
at java.sql.DriverManager.getConnection(DriverManager.java:664) at
java.sql.DriverManager.getConnection(DriverManager.java:247) at
org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at
org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more Caused by: java.security.PrivilegedActionException:
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
com.microsoft.sqlserver.jdbc.KerbAuthentication.getClientCredential(KerbAuthentication.java:199)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:150)
... 26 more Caused by: GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt) at
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at
sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193)
at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) at
sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:62) at
sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:189)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:187)
... 30 more
Does anyone know if Scoop support forwarding the Kerberos ticket? and if
so, what do we need to configure to have it do so? Any comments on how
this should work, or websites discussing how you got this working in your
environment?
There is a Hortonworks Community post on this as well:
https://community.hortonworks.com/questions/52137/sqlserver-sqoop-import-integrated-security-kerbero.html
I would like to verify each step of the process can support Kerberos, and
then verify the configurations to determine if there is any configuration
set that might prevent this from working.
Any help would be appreciated.
thanks.
Jonathan Kaufman | DevOps Engineer | Foot Locker Inc.
(414-357-4062) | jkaufman@footlocker.com
Visit us on-line at footlocker.com.
The information in this e-mail, and any attachment therein, is
confidential and for use by the addressee only. If you are not the
intended recipient, please return the e-mail to the sender and delete it
from your computer. Although the Company attempts to sweep e-mail and
attachments for viruses, it does not guarantee that either are virus-free
and accepts no liability for any damage sustained as a result of viruses.
Re: Help using Scoop 1.4.6 with Kerberos Authentication on Import with
MapReduce.
Posted by Jonathan Kaufman <jk...@footlocker.com>.
Could it be a requirement of the jdbc driver, as this doesn't work without
using the integratedSecurity and authenticationScheme options.
[USER@FQDN~]$ klist -f
Ticket cache: FILE:/tmp/krb5cc_cdc478262072_7Vemq9
Default principal: USER@DOMAIN
Valid starting Expires Service principal
09/08/2016 16:51:36 09/09/2016 02:51:36 krbtgt/DOMAIN@DOMAIN
renew until 09/15/2016 16:51:36, Flags: FRIA
[USER@FQDN~]$ sqoop list-databases --connect "jdbc:sqlserver://FQDN"
--username 'DOMAIN\USER'
Warning: /usr/hdp/2.4.2.0-258/hbase does not exist! HBase imports will
fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/hdp/2.4.2.0-258/accumulo does not exist! Accumulo imports
will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/09/08 16:54:56 INFO sqoop.Sqoop: Running Sqoop version:
1.4.6.2.4.2.0-258
16/09/08 16:54:56 INFO manager.SqlManager: Using default fetchSize of 1000
16/09/08 16:54:56 ERROR manager.CatalogQueryManager: Failed to list
databases
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user
'DOMAIN\USER'. ClientConnectionId:eb2fb32a-60be-4e92-9eaa-3fcf79fe95a7
at
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:217)
at
com.microsoft.sqlserver.jdbc.TDSTokenHandler.onEOF(tdsparser.java:279)
at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:99)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4343)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3159)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.access$100(SQLServerConnection.java:42)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3122)
at
com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1980)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1627)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1458)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:772)
at
com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1168)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at
org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:885)
at
org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at
org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
at
org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
16/09/08 16:54:56 ERROR sqoop.Sqoop: Got exception running Sqoop:
java.lang.RuntimeException:
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user
'DOMAIN\USER'. ClientConnectionId:eb2fb32a-60be-4e92-9eaa-3fcf79fe95a7
java.lang.RuntimeException:
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user
'DOMAIN\USER'. ClientConnectionId:eb2fb32a-60be-4e92-9eaa-3fcf79fe95a7
at
org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:73)
at
org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Login failed
for user 'DOMAIN\USER'.
ClientConnectionId:eb2fb32a-60be-4e92-9eaa-3fcf79fe95a7
at
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:217)
at
com.microsoft.sqlserver.jdbc.TDSTokenHandler.onEOF(tdsparser.java:279)
at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:99)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4343)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3159)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.access$100(SQLServerConnection.java:42)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3122)
at
com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1980)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1627)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1458)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:772)
at
com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1168)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at
org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:885)
at
org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at
org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
... 7 more
[u190491a@mlwhdp1g002 ~]$ sqoop list-databases --connect
"jdbc:sqlserver://FQDN;integratedSecurity=true;authenticationScheme=JavaKerberos"
--username 'DOMAIN\USER'
Warning: /usr/hdp/2.4.2.0-258/hbase does not exist! HBase imports will
fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/hdp/2.4.2.0-258/accumulo does not exist! Accumulo imports
will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/09/08 16:55:05 INFO sqoop.Sqoop: Running Sqoop version:
1.4.6.2.4.2.0-258
16/09/08 16:55:05 INFO manager.SqlManager: Using default fetchSize of 1000
...
Jonathan Kaufman | DevOps Engineer | Foot Locker Inc.
(414-357-4062) | jkaufman@footlocker.com
From: billy iftissen <bi...@gmail.com>
To: user@sqoop.apache.org
Date: 09/08/2016 05:11 AM
Subject: Re: Help using Scoop 1.4.6 with Kerberos Authentication on
Import with MapReduce.
Hello,
Sqoop 1.4.6 support kerberos ==> it mean your mapreduce process is
lanched by a principal that is in realm of KDC (entry point) , verify
ticket with klist
so you dont need to spécify options
integratedSecurity=true;authenticationScheme=JavaKerberos;
hope this helps you
sqoop import --connect "jdbc:sqlserver://FQDN;database=database_name"
--username 'Domain\user' --query 'select * from table where $CONDITIONS'
--split-by Abc_Number --target-dir /user/user/SQL
2016-09-07 17:47 GMT+02:00 Jonathan Kaufman <jk...@footlocker.com>:
Hello all,
I apologize in advance if this seems like a silly question, however I've
not been able to determine if Scoop supports Kerberos delegation on
import, down to the MapReduce process.
I have a user who has a valid Kerberos ticket that supports forwarding,
additionally they are working on a completely kerberized Hadoop cluster
and all hosts in the cluster are trusted for kerberos delegation.
We have been able to use Scoop with Kerberos Authentication when
connecting to a Microsoft SQL Server using the following command:
sqoop
list-databases --connect
"jdbc:sqlserver://hostname.domain.com
;integratedSecurity=true;authenticationScheme=JavaKerberos"
--username 'DOMAIN\USER'
So we tried to do an import using Kerberos, but it has failed with an
error that indicates that MapReduce does not have a kerberos ticket.
Here is the command used on the Sqoop import.
sqoop import --connect
"jdbc:sqlserver://FQDN;integratedSecurity=true;authenticationScheme=JavaKerberos;database=database_name"
--username 'Domain\user' --query 'select * from table where $CONDITIONS'
--split-by Abc_Number --target-dir /user/user/Sql
Here is the error:
INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service:
ha-hdfs:NNHA, Ident: (HDFS_DELEGATION_TOKEN token 2496 for ) INFO
impl.YarnClientImpl: Submitted application INFO mapreduce.Job: The url to
track the job: INFO mapreduce.Job: Running job: job_1 INFO mapreduce.Job:
Job job_1 running in uber mode : false INFO mapreduce.Job: map 0% reduce
0% INFO mapreduce.Job: Task Id : attempt_1470416754637_0238_m_000003_0,
Status : FAILED Error: java.lang.RuntimeException:
java.lang.RuntimeException:
com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication
failed. ClientConnectionId:366c38c7-c at
org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by:
java.lang.RuntimeException:
com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication
failed. ClientConnectionId:366c38c7-c at
org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at
org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more Caused by: com.microsoft.sqlserver.jdbc.SQLServerException:
Integrated authentication failed. ClientConnectionId:366c38c7-c at
com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:176)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication.GenerateClientContext(KerbAuthentication.java:296)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4084)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3159)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.access$100(SQLServerConnection.java:42)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3122)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505) at
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1980)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1627)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1458)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:772)
at
com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1168)
at java.sql.DriverManager.getConnection(DriverManager.java:664) at
java.sql.DriverManager.getConnection(DriverManager.java:247) at
org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at
org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more Caused by: java.security.PrivilegedActionException:
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
com.microsoft.sqlserver.jdbc.KerbAuthentication.getClientCredential(KerbAuthentication.java:199)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:150)
... 26 more Caused by: GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt) at
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at
sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193)
at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) at
sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:62) at
sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:189)
at
com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:187)
... 30 more
Does anyone know if Scoop support forwarding the Kerberos ticket? and if
so, what do we need to configure to have it do so? Any comments on how
this should work, or websites discussing how you got this working in your
environment?
There is a Hortonworks Community post on this as well:
https://community.hortonworks.com/questions/52137/sqlserver-sqoop-import-integrated-security-kerbero.html
I would like to verify each step of the process can support Kerberos, and
then verify the configurations to determine if there is any configuration
set that might prevent this from working.
Any help would be appreciated.
thanks.
Jonathan Kaufman | DevOps Engineer | Foot Locker Inc.
(414-357-4062) | jkaufman@footlocker.com
Visit us on-line at footlocker.com.
The information in this e-mail, and any attachment therein, is
confidential and for use by the addressee only. If you are not the
intended recipient, please return the e-mail to the sender and delete it
from your computer. Although the Company attempts to sweep e-mail and
attachments for viruses, it does not guarantee that either are virus-free
and accepts no liability for any damage sustained as a result of viruses.
--
Rejoignez moi sur le réseau professionnel viadeo à l'adresse:
http://www.viadeo.com/fr/profile/matouk.iftissen1
Re: Help using Scoop 1.4.6 with Kerberos Authentication on Import
with MapReduce.
Posted by billy iftissen <bi...@gmail.com>.
Hello,
Sqoop 1.4.6 support kerberos ==> it mean your mapreduce process is lanched
by a *principal *that is in realm of KDC (entry point) , verify ticket with
klist
so you dont need to spécify options integratedSecurity=true;
authenticationScheme=JavaKerberos;
hope this helps you
sqoop import --connect "jdbc:sqlserver://FQDN;database=database_name"
--username 'Domain\user' --query 'select * from table where $CONDITIONS'
--split-by Abc_Number --target-dir /user/user/SQL
2016-09-07 17:47 GMT+02:00 Jonathan Kaufman <jk...@footlocker.com>:
> Hello all,
>
> I apologize in advance if this seems like a silly question, however I've
> not been able to determine if Scoop supports Kerberos delegation on
> import, down to the MapReduce process.
>
> I have a user who has a valid Kerberos ticket that supports forwarding,
> additionally they are working on a completely kerberized Hadoop cluster and
> all hosts in the cluster are trusted for kerberos delegation.
>
> We have been able to use Scoop with Kerberos Authentication when
> connecting to a Microsoft SQL Server using the following command:
>
> sqoop
> list-databases --connect
> "jdbc:sqlserver://hostname.domain.com;integratedSecurity=
> true;authenticationScheme=JavaKerberos"
> --username 'DOMAIN\USER'
>
> So we tried to do an import using Kerberos, but it has failed with an
> error that indicates that MapReduce does not have a kerberos ticket.
>
> Here is the command used on the Sqoop import.
>
> sqoop import --connect "jdbc:sqlserver://FQDN;integratedSecurity=true;
> authenticationScheme=JavaKerberos;database=database_name" --username
> 'Domain\user' --query 'select * from table where $CONDITIONS' --split-by
> Abc_Number --target-dir /user/user/Sql
>
> Here is the error:
> INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service:
> ha-hdfs:NNHA, Ident: (HDFS_DELEGATION_TOKEN token 2496 for ) INFO
> impl.YarnClientImpl: Submitted application INFO mapreduce.Job: The url to
> track the job: INFO mapreduce.Job: Running job: job_1 INFO mapreduce.Job:
> Job job_1 running in uber mode : false INFO mapreduce.Job: map 0% reduce 0%
> INFO mapreduce.Job: Task Id : attempt_1470416754637_0238_m_000003_0,
> Status : FAILED Error: java.lang.RuntimeException:
> java.lang.RuntimeException: com.microsoft.sqlserver.jdbc.SQLServerException:
> Integrated authentication failed. ClientConnectionId:366c38c7-c at
> org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1709) at org.apache.hadoop.mapred.
> YarnChild.main(YarnChild.java:162) Caused by: java.lang.RuntimeException:
> com.microsoft.sqlserver.jdbc.SQLServerException: Integrated
> authentication failed. ClientConnectionId:366c38c7-c at
> org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
> at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
> ... 9 more Caused by: com.microsoft.sqlserver.jdbc.SQLServerException:
> Integrated authentication failed. ClientConnectionId:366c38c7-c at
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399)
> at com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:176)
> at com.microsoft.sqlserver.jdbc.KerbAuthentication.GenerateClientContext(KerbAuthentication.java:296)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4084)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3159)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.access$
> 100(SQLServerConnection.java:42) at com.microsoft.sqlserver.jdbc.
> SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3122)
> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505) at
> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1980)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1627)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1458)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:772)
> at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1168)
> at java.sql.DriverManager.getConnection(DriverManager.java:664) at
> java.sql.DriverManager.getConnection(DriverManager.java:247) at
> org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
> at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
> ... 10 more Caused by: java.security.PrivilegedActionException:
> GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt) at java.security.AccessController.doPrivileged(Native
> Method) at javax.security.auth.Subject.doAs(Subject.java:422) at
> com.microsoft.sqlserver.jdbc.KerbAuthentication.getClientCredential(KerbAuthentication.java:199)
> at com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:150)
> ... 26 more Caused by: GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt) at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193)
> at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) at
> sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:62) at
> sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154)
> at com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:189)
> at com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:187)
> ... 30 more
>
>
> Does anyone know if Scoop support forwarding the Kerberos ticket? and if
> so, what do we need to configure to have it do so? Any comments on how this
> should work, or websites discussing how you got this working in your
> environment?
>
> There is a Hortonworks Community post on this as well: https://community.
> hortonworks.com/questions/52137/sqlserver-sqoop-import-
> integrated-security-kerbero.html
>
> I would like to verify each step of the process can support Kerberos, and
> then verify the configurations to determine if there is any configuration
> set that might prevent this from working.
>
> Any help would be appreciated.
>
> thanks.
>
> Jonathan Kaufman | DevOps Engineer | Foot Locker Inc.
> (414-357-4062) | jkaufman@footlocker.com
>
> Visit us on-line at *footlocker.com*
> <http://www.footlocker.com/home/default.cfm?SID=8431>.
>
>
> The information in this e-mail, and any attachment therein, is
> confidential and for use by the addressee only. If you are not the intended
> recipient, please return the e-mail to the sender and delete it from your
> computer. Although the Company attempts to sweep e-mail and attachments for
> viruses, it does not guarantee that either are virus-free and accepts no
> liability for any damage sustained as a result of viruses.
--
Rejoignez moi sur le réseau professionnel viadeo à l'adresse:
http://www.viadeo.com/fr/profile/matouk.iftissen1