[jira] [Commented] (SQOOP-1168) Sqoop2: Incremental From ( formerly called Incremental Import )

["Vinoth Chandar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SQOOP-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218645#comment-14218645 ] 

Vinoth Chandar commented on SQOOP-1168:
---------------------------------------

Thanks for the update, [~vybs] . Sounds promising! 

>>The latter is more work when writing the same to "HDFS" like data source, since we have to scan all records that have been written before and then modify them,
-- My understanding is that sqoop1 simply produced a new file and left this reconciliation to any further post processing and we want sqoop itself to handle this now? 
-- Also, some jobs might be consuming the records written in the previous run at the same time, we are doing the reconciliation, and thus end up reading some non-existent intermediate state.. Something to hash out? 

One detail I realized on the From/To StateObjects is that they have to be per job?

> Sqoop2: Incremental From ( formerly called Incremental Import )
> ---------------------------------------------------------------
>
>                 Key: SQOOP-1168
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1168
>             Project: Sqoop
>          Issue Type: Bug
>            Reporter: Hari Shreedharan
>            Assignee: Veena Basavaraj
>             Fix For: 1.99.5
>
>
> Initial plan is to follow roughly the same design as Sqoop 1, except provide pluggability to start this through a REST API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)