[jira] [Commented] (FALCON-2234) Falcon feed fails for RDBMS import if extract type is incremental

["Ajay Yadava (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FALCON-2234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15804728#comment-15804728 ] 

Ajay Yadava commented on FALCON-2234:
-------------------------------------

Hello Prashant,

Thank you for reporting this issue. If I recall correctly, the original plan was to add support for [incremental, append] as another patch which never came through and hence has landed the documentation and validation in an inconsistent state. 

I believe the documentation needs to be fixed. Would you like to provide a patch?

> Falcon feed fails for RDBMS import if extract type is incremental
> -----------------------------------------------------------------
>
>                 Key: FALCON-2234
>                 URL: https://issues.apache.org/jira/browse/FALCON-2234
>             Project: Falcon
>          Issue Type: Bug
>          Components: falcon-ui
>    Affects Versions: 0.10
>            Reporter: Prashant
>            Priority: Critical
>
> Falcon feed is failing with org.apache.falcon.entity.parser.ValidationException
> when extract type is incremental.
> As per the Apache doc Falcon supports 
> Import policy
> The valid combinations are:
> [full,snapshot] - data is extracted in full and dumped into the feed instance location.
> [incremental, append] - data is extracted incrementally using the key specified in the deltacolumn
> As per the below code , it appears we only check [full,snapshot] and [incremental, append]  goes to exception 
> https://github.com/apache/falcon/blob/0.10/common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java
> Snippet 
> {code}
> /**
>      * Validate extraction and merge type combination. Currently supported combo:
>      *
>      * ExtractionType = FULL and MergeType = SNAPSHOT.
>      * ExtractionType = INCREMENTAL and MergeType = APPEND.
>      *
>      * @param feed Feed entity
>      * @param cluster Cluster referenced in the Feed definition
>      * @throws FalconException
>      */
>     private void validateFeedExtractionType(Feed feed, Cluster cluster) throws FalconException {
>         Extract extract = cluster.getImport().getSource().getExtract();
>         if (ExtractMethod.FULL == extract.getType())  {
>             if ((MergeType.SNAPSHOT != extract.getMergepolicy())
>                     || (extract.getDeltacolumn() != null)) {
>                 throw new ValidationException(String.format("Feed %s is using FULL "
>                         + "extract method but specifies either a superfluous "
>                         + "deltacolumn  or a mergepolicy other than snapshot", feed.getName()));
>             }
>         }  else {
>             throw new ValidationException(String.format("Feed %s is using unsupported "
>                     + "extraction mechanism %s", feed.getName(), extract.getType().value()));
>         }
>     }
> {code}
> It must have a clause to also check [incremental, append]  .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)