You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Jarek Cecho <ja...@apache.org> on 2016/02/04 00:07:12 UTC
Re: Review Request 41971: Sqoop2: Enable kerberos for
HadoopMiniCluster
> On Jan. 6, 2016, 4:07 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java, line 74
> > <https://reviews.apache.org/r/41971/diff/1/?file=1183938#file1183938line74>
> >
> > Why do we need to change the directory to /tmp/ ?
> >
> > Aren't we by mistake using this for local path somewhere?
>
> Dian Fu wrote:
> In kerberos enabled cluster, we will get exceptions like this with "/mapreduce-job-io":
> ```
> 2016-01-07 10:00:46,381 INFO [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$DefaultAuditLogger.logAuditMessage(FSNamesystem.java:9332)] allowed=false ugi=sqoopclient (auth:TOKEN) via hadoop/bdpe20.sh.intel.com@EXAMPLE.COM (auth:TOKEN) ip=/127.0.0.1 cmd=create src=/mapreduce-job-io/org.apache.sqoop.integration.connector.hdfs.AppendModeTest/test/.577fcfcd-6ef8-491c-ad6c-c6f4983dfdf0/22200b07-c11b-4c39-b0f3-c8236b34f44e.txt dst=null perm=null proto=rpc
> 2016-01-07 10:00:46,381 DEBUG [org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:634)] Served: create queueTime= 1 procesingTime= 1 exception= AccessControlException
> 2016-01-07 10:00:46,381 DEBUG [org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1632)] PrivilegedActionException as:sqoopclient (auth:TOKEN) via hadoop/bdpe20.sh.intel.com@EXAMPLE.COM (auth:TOKEN) cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=sqoopclient, access=WRITE, inode="/":root:supergroup:drwxr-xr-x
> ```
> This reason to this exception is that MR job tries to create directory "/mapreduce-job-io" with user **sqoopclient**. But only superuser is allowed to create directory under "/". While I found a better way to handle this: just create directory "mapreduce-job-io" with superuser and changes its permission to "0777" during cluster initial phase.
Got it, I'm fine with /tmp/ then :)
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41971/#review113083
-----------------------------------------------------------
On Jan. 29, 2016, 8:46 a.m., Dian Fu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41971/
> -----------------------------------------------------------
>
> (Updated Jan. 29, 2016, 8:46 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-2744
> https://issues.apache.org/jira/browse/SQOOP-2744
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> The aim of this JIRA is to enable kerberos for HadoopMiniCluster.
>
>
> Diffs
> -----
>
> test/pom.xml 644a9c7
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java 2c0c4e6
> test/src/main/java/org/apache/sqoop/test/infrastructure/SqoopTestCase.java 74fe29b
> test/src/main/java/org/apache/sqoop/test/kdc/KdcRunner.java aa5c6fc
> test/src/main/java/org/apache/sqoop/test/kdc/MiniKdcRunner.java 299fd9f
> test/src/main/java/org/apache/sqoop/test/kdc/NoKerberosKdcRunner.java 29a005e
> test/src/main/java/org/apache/sqoop/test/minicluster/SqoopMiniCluster.java 4ff97e7
> test/src/test/java/org/apache/sqoop/integration/connector/hdfs/OutputDirectoryTest.java e5e3d26
> test/src/test/java/org/apache/sqoop/integration/connector/hive/FromRDBMSToKiteHiveTest.java ec9f733
> test/src/test/java/org/apache/sqoop/integration/connectorloading/BlacklistedConnectorTest.java 1c7a483
> test/src/test/java/org/apache/sqoop/integration/connectorloading/ClasspathTest.java 5e49064
> test/src/test/java/org/apache/sqoop/integration/connectorloading/ConnectorClasspathIsolationTest.java bbbf749
> test/src/test/resources/hive-tests-suite.xml PRE-CREATION
> test/src/test/resources/integration-tests-suite.xml 1cf3299
>
> Diff: https://reviews.apache.org/r/41971/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Dian Fu
>
>