You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Jainendra Rai <ja...@gmail.com> on 2016/06/01 08:11:52 UTC
Falcon Incremental Import support not available
Hi Team,
We were trying to run an Import job with Apache Falcon. The document says
that
- [incremental, append] - data is extracted incrementally using the key
specified in the deltacolumn is supported
<import>
<source name="mysql-db" tableName="simple">
<extract type="incremental">
<deltacolumn>modified_time</deltacolumn>
<mergepolicy>append</mergepolicy>
</extract>
<fields>
<includes>
<field>id</field>
<field>name</field>
</includes>
</fields>
</source>
<arguments>
<argument name="--split-by" value="id"/>
<argument name="--num-mappers" value="2"/>
</arguments>
</import>
Whille [incremental, snapshot] - this is not supported.
We were trying to work with the former one and suprisingly when we submit
the feed we get an Error
ERROR: Bad Request;default/Feed feed-employee2 is using unsupported
extraction mechanism incremental
Log:
2016-06-01 08:04:24,761 INFO - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~
All locks released on feed-employee2 (AbstractEntityManager:371)
2016-06-01 08:04:24,762 ERROR - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~
Unable to persist entity object (AbstractEntityManager:194)
org.apache.falcon.entity.parser.ValidationException: Feed feed-employee2 is
using unsupported extraction mechanism incremental
at
org.apache.falcon.entity.parser.FeedEntityParser.validateFeedExtractionType(FeedEntityParser.java:595)
at
org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:105)
at
org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:71)
at
org.apache.falcon.resource.AbstractEntityManager.validate(AbstractEntityManager.java:519)
at
org.apache.falcon.resource.AbstractEntityManager.submitInternal(AbstractEntityManager.java:443)
at
org.apache.falcon.resource.AbstractEntityManager.submit(AbstractEntityManager.java:191)
at
org.apache.falcon.resource.ConfigSyncService.submit(ConfigSyncService.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49)
at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$2.doExecute(SchedulableEntityManagerProxy.java:177)
at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:773)
at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit_aroundBody2(SchedulableEntityManagerProxy.java:179)
at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure3.run(SchedulableEntityManagerProxy.java:1)
at
org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at
org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)
at
org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit(SchedulableEntityManagerProxy.java:157)
at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:108)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:188)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:595)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:554)
at
org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:197)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.falcon.security.HostnameFilter.doFilter(HostnameFilter.java:82)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at
org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2016-06-01 08:04:24,763 INFO - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~
{Action:submit, Dimensions:{entityType=feed, colo=NULL}, Status: FAILED,
Time-taken:11824449 ns} (METRIC:38)
2016-06-01 08:04:24,763 ERROR - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~
Exception in sending Notification from
EmailNotificationPlugin:java.lang.NullPointerException
(EmailNotificationPlugin:71)
2016-06-01 08:04:24,764 DEBUG - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:] ~ Audit: hadoop/127.0.0.1 performed
request https://localhost:15443/api/entities/submit/feed (127.0.0.1) at
time 2016-06-01T08:04Z (FalconAuditFilter:86)
2016-06-01 08:04:24,764 INFO - [718278514@qtp-2060207812-55 -
c3a7419c-f8f6-4b93-9bf8-1349ec166407:] ~ Audit: hadoop@127.0.0.1 performed
https://localhost:15443/api/entities/submit/feed (127.0.0.1) at
2016-06-01T08:04Z (AUDIT:48
Should this error come up.
PLease have a look at the submitted feed too.
<?xml version="1.0" encoding="UTF-8"?>
<!--
A feed representing Hourly customer email data retained for 90 days
-->
<feed description="Raw customer email feed" name="feed1"
xmlns="uri:falcon:feed:0.1">
<tags>externalSystem=USWestEmailServers,classification=secure</tags>
<groups>DataImportPipeline</groups>
<frequency>minutes(6)</frequency>
<late-arrival cut-off="hours(4)"/>
<clusters>
<cluster name="primaryCluster" type="source">
<validity start="2016-05-23T00:00Z"
end="2017-05-31T00:00Z"/>
<retention limit="days(90)" action="delete"/>
<import>
<source name="mysql-db" tableName="employee">
<extract type="incremental">
<deltacolumn>modified_item</deltacolumn>
<mergepolicy>append</mergepolicy>
</extract>
</source>
<arguments>
<argument name="--split-by" value="id"/>
<argument name="--num-mappers" value="2"/>
</arguments>
</import>
</cluster>
</clusters>
<locations>
<location type="data"
path="/falcon/${YEAR}-${MONTH}-${DAY}-${HOUR}-${MINUTE}"/>
<location type="stats" path="/none"/>
<location type="meta" path="/none"/>
</locations>
<notification type="email" to="xyz.com"/>
<ACL owner="hadoop" group="hadoop" permission="0755"/>
<schema location="/none" provider="none"/>
</feed>
Re: Falcon Incremental Import support not available
Posted by Ying Zheng <yz...@hortonworks.com>.
Jainendra: The current Falcon doesn’t support the incremental method. It is an up-coming feature.
Thank you.
Ying
From: <Rai>, Jainendra <JR...@beckman.com>>
Reply-To: "dev@falcon.apache.org<ma...@falcon.apache.org>" <de...@falcon.apache.org>>
Date: Wednesday, June 1, 2016 at 6:30 AM
To: "dev@falcon.apache.org<ma...@falcon.apache.org>" <de...@falcon.apache.org>>
Subject: RE: Falcon Incremental Import support not available
Hi Team,
It seems the code could have some errors..
This is the method that throws the error.. The method says that incremental and Mergetype append is supported but then according to this Incremental extract will always throw the exception.
Am I looking into it correct? Please advise
[cid:image003.jpg@01D1BC37.DA3B0820]
Kind Regards
Jainendra
[unnamed]Go Green
From: Jainendra Rai [mailto:jainendra72@gmail.com]
Sent: Wednesday, June 01, 2016 6:56 PM
To: Rai, Jainendra
Subject: Fwd: Falcon Incremental Import support not available
Best Regards,
+91-9741823970
jainendra72@gmail.com<ma...@gmail.com>
---------- Forwarded message ----------
From: Jainendra Rai <ja...@gmail.com>>
Date: Wed, Jun 1, 2016 at 1:41 PM
Subject: Falcon Incremental Import support not available
To: dev@falcon.apache.org<ma...@falcon.apache.org>
Hi Team,
We were trying to run an Import job with Apache Falcon. The document says that
· [incremental, append] - data is extracted incrementally using the key specified in the deltacolumn is supported
<import>
<source name="mysql-db" tableName="simple">
<extract type="incremental">
<deltacolumn>modified_time</deltacolumn>
<mergepolicy>append</mergepolicy>
</extract>
<fields>
<includes>
<field>id</field>
<field>name</field>
</includes>
</fields>
</source>
<arguments>
<argument name="--split-by" value="id"/>
<argument name="--num-mappers" value="2"/>
</arguments>
</import>
Whille [incremental, snapshot] - this is not supported.
We were trying to work with the former one and suprisingly when we submit the feed we get an Error
ERROR: Bad Request;default/Feed feed-employee2 is using unsupported extraction mechanism incremental
Log:
2016-06-01 08:04:24,761 INFO - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~ All locks released on feed-employee2 (AbstractEntityManager:371)
2016-06-01 08:04:24,762 ERROR - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~ Unable to persist entity object (AbstractEntityManager:194)
org.apache.falcon.entity.parser.ValidationException: Feed feed-employee2 is using unsupported extraction mechanism incremental
at org.apache.falcon.entity.parser.FeedEntityParser.validateFeedExtractionType(FeedEntityParser.java:595)
at org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:105)
at org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:71)
at org.apache.falcon.resource.AbstractEntityManager.validate(AbstractEntityManager.java:519)
at org.apache.falcon.resource.AbstractEntityManager.submitInternal(AbstractEntityManager.java:443)
at org.apache.falcon.resource.AbstractEntityManager.submit(AbstractEntityManager.java:191)
at org.apache.falcon.resource.ConfigSyncService.submit(ConfigSyncService.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$2.doExecute(SchedulableEntityManagerProxy.java:177)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:773)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit_aroundBody2(SchedulableEntityManagerProxy.java:179)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure3.run(SchedulableEntityManagerProxy.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit(SchedulableEntityManagerProxy.java:157)
at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:108)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:188)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:595)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:554)
at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:197)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.falcon.security.HostnameFilter.doFilter(HostnameFilter.java:82)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2016-06-01 08:04:24,763 INFO - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~ {Action:submit, Dimensions:{entityType=feed, colo=NULL}, Status: FAILED, Time-taken:11824449 ns} (METRIC:38)
2016-06-01 08:04:24,763 ERROR - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~ Exception in sending Notification from EmailNotificationPlugin:java.lang.NullPointerException (EmailNotificationPlugin:71)
2016-06-01 08:04:24,764 DEBUG - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:] ~ Audit: hadoop/127.0.0.1<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=AuZ2Uqn6QXWYp_ESwW_1AMsG9fhQCT1olHCBR6HPoRg&e=> performed request https://localhost:15443/api/entities/submit/feed<https://urldefense.proofpoint.com/v2/url?u=https-3A__localhost-3A15443_api_entities_submit_feed&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=jKPtKEZZX76LCp06hGh_Vlv4AbotHIotULACNZmpbUE&e=> (127.0.0.1) at time 2016-06-01T08:04Z (FalconAuditFilter:86)
2016-06-01 08:04:24,764 INFO - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:] ~ Audit: hadoop@127.0.0.1<ma...@127.0.0.1> performed https://localhost:15443/api/entities/submit/feed<https://urldefense.proofpoint.com/v2/url?u=https-3A__localhost-3A15443_api_entities_submit_feed&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=jKPtKEZZX76LCp06hGh_Vlv4AbotHIotULACNZmpbUE&e=> (127.0.0.1) at 2016-06-01T08:04Z (AUDIT:48
Should this error come up.
PLease have a look at the submitted feed too.
<?xml version="1.0" encoding="UTF-8"?>
<!--
A feed representing Hourly customer email data retained for 90 days
-->
<feed description="Raw customer email feed" name="feed1" xmlns="uri:falcon:feed:0.1">
<tags>externalSystem=USWestEmailServers,classification=secure</tags>
<groups>DataImportPipeline</groups>
<frequency>minutes(6)</frequency>
<late-arrival cut-off="hours(4)"/>
<clusters>
<cluster name="primaryCluster" type="source">
<validity start="2016-05-23T00:00Z" end="2017-05-31T00:00Z"/>
<retention limit="days(90)" action="delete"/>
<import>
<source name="mysql-db" tableName="employee">
<extract type="incremental">
<deltacolumn>modified_item</deltacolumn>
<mergepolicy>append</mergepolicy>
</extract>
</source>
<arguments>
<argument name="--split-by" value="id"/>
<argument name="--num-mappers" value="2"/>
</arguments>
</import>
</cluster>
</clusters>
<locations>
<location type="data" path="/falcon/${YEAR}-${MONTH}-${DAY}-${HOUR}-${MINUTE}"/>
<location type="stats" path="/none"/>
<location type="meta" path="/none"/>
</locations>
<notification type="email" to="xyz.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__xyz.com&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=0bHHCBjZnnogp83kfVwhfyxKkEWQizicw7CrGfPklGI&e=>"/>
<ACL owner="hadoop" group="hadoop" permission="0755"/>
<schema location="/none" provider="none"/>
</feed>
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
RE: Falcon Incremental Import support not available
Posted by "Rai, Jainendra" <JR...@beckman.com>.
Hi Team,
It seems the code could have some errors..
This is the method that throws the error.. The method says that incremental and Mergetype append is supported but then according to this Incremental extract will always throw the exception.
Am I looking into it correct? Please advise
[cid:image003.jpg@01D1BC37.DA3B0820]
Kind Regards
Jainendra
[unnamed] Go Green
From: Jainendra Rai [mailto:jainendra72@gmail.com]
Sent: Wednesday, June 01, 2016 6:56 PM
To: Rai, Jainendra
Subject: Fwd: Falcon Incremental Import support not available
Best Regards,
+91-9741823970
jainendra72@gmail.com<ma...@gmail.com>
---------- Forwarded message ----------
From: Jainendra Rai <ja...@gmail.com>>
Date: Wed, Jun 1, 2016 at 1:41 PM
Subject: Falcon Incremental Import support not available
To: dev@falcon.apache.org<ma...@falcon.apache.org>
Hi Team,
We were trying to run an Import job with Apache Falcon. The document says that
· [incremental, append] - data is extracted incrementally using the key specified in the deltacolumn is supported
<import>
<source name="mysql-db" tableName="simple">
<extract type="incremental">
<deltacolumn>modified_time</deltacolumn>
<mergepolicy>append</mergepolicy>
</extract>
<fields>
<includes>
<field>id</field>
<field>name</field>
</includes>
</fields>
</source>
<arguments>
<argument name="--split-by" value="id"/>
<argument name="--num-mappers" value="2"/>
</arguments>
</import>
Whille [incremental, snapshot] - this is not supported.
We were trying to work with the former one and suprisingly when we submit the feed we get an Error
ERROR: Bad Request;default/Feed feed-employee2 is using unsupported extraction mechanism incremental
Log:
2016-06-01 08:04:24,761 INFO - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~ All locks released on feed-employee2 (AbstractEntityManager:371)
2016-06-01 08:04:24,762 ERROR - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~ Unable to persist entity object (AbstractEntityManager:194)
org.apache.falcon.entity.parser.ValidationException: Feed feed-employee2 is using unsupported extraction mechanism incremental
at org.apache.falcon.entity.parser.FeedEntityParser.validateFeedExtractionType(FeedEntityParser.java:595)
at org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:105)
at org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:71)
at org.apache.falcon.resource.AbstractEntityManager.validate(AbstractEntityManager.java:519)
at org.apache.falcon.resource.AbstractEntityManager.submitInternal(AbstractEntityManager.java:443)
at org.apache.falcon.resource.AbstractEntityManager.submit(AbstractEntityManager.java:191)
at org.apache.falcon.resource.ConfigSyncService.submit(ConfigSyncService.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$2.doExecute(SchedulableEntityManagerProxy.java:177)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:773)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit_aroundBody2(SchedulableEntityManagerProxy.java:179)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure3.run(SchedulableEntityManagerProxy.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit(SchedulableEntityManagerProxy.java:157)
at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:108)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:188)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:595)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:554)
at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:197)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.falcon.security.HostnameFilter.doFilter(HostnameFilter.java:82)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2016-06-01 08:04:24,763 INFO - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~ {Action:submit, Dimensions:{entityType=feed, colo=NULL}, Status: FAILED, Time-taken:11824449 ns} (METRIC:38)
2016-06-01 08:04:24,763 ERROR - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed] ~ Exception in sending Notification from EmailNotificationPlugin:java.lang.NullPointerException (EmailNotificationPlugin:71)
2016-06-01 08:04:24,764 DEBUG - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:] ~ Audit: hadoop/127.0.0.1<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=AuZ2Uqn6QXWYp_ESwW_1AMsG9fhQCT1olHCBR6HPoRg&e=> performed request https://localhost:15443/api/entities/submit/feed<https://urldefense.proofpoint.com/v2/url?u=https-3A__localhost-3A15443_api_entities_submit_feed&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=jKPtKEZZX76LCp06hGh_Vlv4AbotHIotULACNZmpbUE&e=> (127.0.0.1) at time 2016-06-01T08:04Z (FalconAuditFilter:86)
2016-06-01 08:04:24,764 INFO - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:] ~ Audit: hadoop@127.0.0.1<ma...@127.0.0.1> performed https://localhost:15443/api/entities/submit/feed<https://urldefense.proofpoint.com/v2/url?u=https-3A__localhost-3A15443_api_entities_submit_feed&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=jKPtKEZZX76LCp06hGh_Vlv4AbotHIotULACNZmpbUE&e=> (127.0.0.1) at 2016-06-01T08:04Z (AUDIT:48
Should this error come up.
PLease have a look at the submitted feed too.
<?xml version="1.0" encoding="UTF-8"?>
<!--
A feed representing Hourly customer email data retained for 90 days
-->
<feed description="Raw customer email feed" name="feed1" xmlns="uri:falcon:feed:0.1">
<tags>externalSystem=USWestEmailServers,classification=secure</tags>
<groups>DataImportPipeline</groups>
<frequency>minutes(6)</frequency>
<late-arrival cut-off="hours(4)"/>
<clusters>
<cluster name="primaryCluster" type="source">
<validity start="2016-05-23T00:00Z" end="2017-05-31T00:00Z"/>
<retention limit="days(90)" action="delete"/>
<import>
<source name="mysql-db" tableName="employee">
<extract type="incremental">
<deltacolumn>modified_item</deltacolumn>
<mergepolicy>append</mergepolicy>
</extract>
</source>
<arguments>
<argument name="--split-by" value="id"/>
<argument name="--num-mappers" value="2"/>
</arguments>
</import>
</cluster>
</clusters>
<locations>
<location type="data" path="/falcon/${YEAR}-${MONTH}-${DAY}-${HOUR}-${MINUTE}"/>
<location type="stats" path="/none"/>
<location type="meta" path="/none"/>
</locations>
<notification type="email" to="xyz.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__xyz.com&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=0bHHCBjZnnogp83kfVwhfyxKkEWQizicw7CrGfPklGI&e=>"/>
<ACL owner="hadoop" group="hadoop" permission="0755"/>
<schema location="/none" provider="none"/>
</feed>
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.