You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "davide anastasia (JIRA)" <ji...@apache.org> on 2015/07/06 11:29:04 UTC
[jira] [Commented] (OOZIE-2292) Oozie doesn't read S3 credentials
from core-site.xml
[ https://issues.apache.org/jira/browse/OOZIE-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14614767#comment-14614767 ]
davide anastasia commented on OOZIE-2292:
-----------------------------------------
To add more details to this ticket, I have put the S3 credential into the hadoop-conf/core-site.xml of my Oozie installation. As I am using Amazon Elastic MapReduce (EMR), I don't have S3 credential into the core-site.xml file of my Hadoop cluster (EMR uses EmrFs, which is a custom FileSystem implementation by Amazon for S3). How do the two interact?
> Oozie doesn't read S3 credentials from core-site.xml
> ----------------------------------------------------
>
> Key: OOZIE-2292
> URL: https://issues.apache.org/jira/browse/OOZIE-2292
> Project: Oozie
> Issue Type: Bug
> Components: core
> Affects Versions: 4.2.0
> Reporter: davide anastasia
>
> I have compiled Oozie 4.2.0 using Hadoop 2.4.0. I then changed my configuration as explained in OOZIE-426. My coordinator/workflow XML(s) are on HDFS, while my input data is in S3. Ideally I would have loved to have my coordinator/workflow XML(s) on S3 too.
> When I perform a dry run, I get the following:
> {code}
> 2015-07-03 16:56:16,639 WARN CoordSubmitXCommand:523 - SERVER[englebert] USER[davide] GROUP[-] TOKEN[] APP[my-app-coord] JOB[0000000-150703165554194-oozie-davi-C] ACTION[-] ERROR:
> java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties (respectively).
> at org.apache.hadoop.fs.s3.S3Credentials.initialize(S3Credentials.java:70)
> at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at org.apache.hadoop.fs.s3native.$Proxy28.initialize(Unknown Source)
> at org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:272)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
> at org.apache.oozie.service.HadoopAccessorService$4.run(HadoopAccessorService.java:488)
> at org.apache.oozie.service.HadoopAccessorService$4.run(HadoopAccessorService.java:486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:486)
> at org.apache.oozie.dependency.FSURIHandler.getFileSystem(FSURIHandler.java:166)
> at org.apache.oozie.dependency.FSURIHandler.exists(FSURIHandler.java:100)
> at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:579)
> at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkListOfPaths(CoordActionInputCheckXCommand.java:550)
> at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkResolvedUris(CoordActionInputCheckXCommand.java:520)
> at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkInput(CoordActionInputCheckXCommand.java:351)
> at org.apache.oozie.command.coord.CoordCommandUtils.dryRunCoord(CoordCommandUtils.java:573)
> at org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(CoordCommandUtils.java:541)
> at org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materializeActions(CoordMaterializeTransitionXCommand.java:485)
> at org.apache.oozie.command.coord.CoordSubmitXCommand.getDryRun(CoordSubmitXCommand.java:316)
> at org.apache.oozie.command.coord.CoordSubmitXCommand.submitJob(CoordSubmitXCommand.java:248)
> at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:203)
> at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:82)
> at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:30)
> at org.apache.oozie.command.XCommand.call(XCommand.java:286)
> at org.apache.oozie.CoordinatorEngine.dryRunSubmit(CoordinatorEngine.java:561)
> at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:228)
> at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:95)
> at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:102)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:643)
> at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:304)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:392)
> at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
> at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:620)
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)