You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by "Ajay Yadava (JIRA)" <ji...@apache.org> on 2015/08/20 21:02:46 UTC
[jira] [Comment Edited] (FALCON-1411) El expressions not getting
evaluated correctly with falcon-0.7 jar
[ https://issues.apache.org/jira/browse/FALCON-1411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14705564#comment-14705564 ]
Ajay Yadava edited comment on FALCON-1411 at 8/20/15 7:02 PM:
--------------------------------------------------------------
On debugging Jaydeep Vishwakarma found that the actual issue is that during dryRun Oozie is actually looking for an instance path which can match to latest(0) (any instance) and it doesn't find any and being a synchronous operation it fails. If a feed instance is created/exists then the issue goes away. We tested this with feeds with existing instances and same consumer process worked smoothly.
was (Author: ajayyadava):
On debugging Jaydeep Vishwakarma found that the actual issue is that during dryRun Oozie is actually looking for an instance path which can match to latest(0) (any instance) and it doesn't find any and being a synchronous operation it fails. If a feed instance is created/exists then the issue goes away. We tested this with feeds with existing instances and same consumer process worked smoothly.
That said it is not always possible to wait for feed instances to be available or bootstrap with a dummy one and this feature will be useful in those cases.
> El expressions not getting evaluated correctly with falcon-0.7 jar
> ------------------------------------------------------------------
>
> Key: FALCON-1411
> URL: https://issues.apache.org/jira/browse/FALCON-1411
> Project: Falcon
> Issue Type: Bug
> Components: oozie
> Affects Versions: 0.7
> Environment: QA
> Reporter: Pragya Mittal
> Assignee: Pallavi Rao
>
> Following El-expressions are broken :
> 1. currentWeek and lastWeek
> While scheduling a process containing currentWeek/lastWeek el expression it gets scheduled but instance keeps on waiting (although there are no missing dependency)
> Process -definition is :
> {code}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <process name="Adda4dfb0-5eb92594" xmlns="uri:falcon:process:0.1">
> <clusters>
> <cluster name="Adda4dfb0-a611389d">
> <validity start="2015-02-17T10:30Z" end="2015-02-17T10:50Z"/>
> </cluster>
> </clusters>
> <parallel>1</parallel>
> <order>FIFO</order>
> <frequency>minutes(5)</frequency>
> <timezone>UTC</timezone>
> <inputs>
> <input name="inputData" feed="Adda4dfb0-04fe993c" start="currentWeek('WED',2,15)" end="currentWeek('WED',2,25)"/>
> </inputs>
> <outputs>
> <output name="outputData" feed="Adda4dfb0-68dd6cce" instance="now(0,0)"/>
> </outputs>
> <properties>
> <property name="queueName" value="default"/>
> <property name="fileTime" value="${formatTime(dateOffset(instanceTime(), 1, 'DAY'), 'yyyy-MMM-dd')}"/>
> <property name="user" value="${user()}"/>
> </properties>
> <workflow path="/tmp/falcon-regression/ELExpCurrentAndLastWeekTest/aggregator"/>
> <retry policy="periodic" delay="minutes(3)" attempts="3"/>
> <ACL owner="pragya" group="dataqa" permission="*"/>
> </process>
> {code}
> Oozie stack-trace is as follows :
> {code}
> 2015-08-19 12:34:57,199 WARN pool-6-thread-2 CallableQueueService$CompositeCallable - SERVER[ip-192-168-138-200] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-150819121241197-oozie-oozi-C] ACTION[0000002-150819121241197-oozie-oozi-C@4] exception callable [coord_action_input], E1021: Coord Action Input Check Error: Unable to evaluate :${dataIn('inputData', 'null')}:
> org.apache.oozie.command.CommandException: E1021: Coord Action Input Check Error: Unable to evaluate :${dataIn('inputData', 'null')}:
> at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:279)
> at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:71)
> at org.apache.oozie.command.XCommand.call(XCommand.java:286)
> at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321)
> at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250)
> at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Unable to evaluate :${dataIn('inputData', 'null')}:
> at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:723)
> at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.resolveTagContents(CoordActionInputCheckXCommand.java:398)
> at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.materializeDataProperties(CoordActionInputCheckXCommand.java:379)
> at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.resolveCoordConfiguration(CoordActionInputCheckXCommand.java:288)
> at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:230)
> ... 8 more
> Caused by: javax.servlet.jsp.el.ELException: An error occurred while evaluating function "dataIn"
> at org.apache.commons.el.Logger.logError(Logger.java:484)
> at org.apache.commons.el.Logger.logError(Logger.java:545)
> at org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:174)
> at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:263)
> at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
> at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:204)
> at org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:714)
> ... 12 more
> {code}
> Oozie coordinator.xml contains the following property :
> {code}
> <property>
> <name>falconInPaths</name>
> <value>${dataIn('inputData', 'null')}</value>
> </property>
> {code}
> 2. future and latest
> While scheduling a process containing future/latest el expression it gets submitted but fails on scheduling.
> Falcon stack trace shows falcon is not able to communicate with oozie and gives socket timeout exception .
> {code}
> 2015-08-19 12:31:02,527 ERROR - [1768191773@qtp-351417028-14 - 25e8c020-17ed-4d56-85e7-f5ded1f6d442:pragya:POST//entities/schedule/process/Ac21dad17-c7ef19c7] ~ Unable to schedule workflow (AbstractSchedulableEntityManager:75)
> org.apache.falcon.FalconException: Entity schedule failed for process: Ac21dad17-c7ef19c7
> at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:95)
> at org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:72)
> at org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:130)
> at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:106)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:184)
> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:396)
> at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:193)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: org.apache.falcon.FalconException: IO_ERROR : java.net.SocketTimeoutException: Read timed out
> at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:223)
> at org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:171)
> at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:93)
> ... 45 more
> Caused by: IO_ERROR : java.net.SocketTimeoutException: Read timed out
> at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:523)
> at org.apache.oozie.client.OozieClient.dryrun(OozieClient.java:735)
> at org.apache.oozie.client.ProxyOozieClient.access$401(ProxyOozieClient.java:48)
> at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:177)
> at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:174)
> at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:198)
> at org.apache.oozie.client.ProxyOozieClient.dryrun(ProxyOozieClient.java:174)
> at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:221)
> ... 47 more
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)