[jira] [Commented] (SQOOP-1803) JobManager and Execution Engine changes: Support for a injecting and pulling out configs and job output in connectors

["Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SQOOP-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14490678#comment-14490678 ] 

Jarek Jarcec Cecho commented on SQOOP-1803:
-------------------------------------------

My apologies for disappearing for a while - I was sadly side tracked by my day job :( I've took a look at your POC and I think that it's very similar to mine in nature [~vybs]. We're both expecting connector developers to prepare state that is then persisted somewhere. My POC was storing it in context classes, whereas you're making it a bit more explicit and storing it in {{MSubmission}} (and hence in our repository). We also differ in a way that connector developer is updating the configuration objects. In my approach, the connector developer is directly updating the objects, whereas it seems that your approach is requesting developer to store the config name in a special map.

I'm wondering if anyone else has took a look at both POCs?

> JobManager and Execution Engine changes: Support for a injecting and pulling out configs and job output in connectors 
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: SQOOP-1803
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1803
>             Project: Sqoop
>          Issue Type: Sub-task
>            Reporter: Veena Basavaraj
>            Assignee: Veena Basavaraj
>             Fix For: 2.0.0
>
>         Attachments: SQOOP-1803-POC-2.patch, SQOOP-1803-POC.patch
>
>
> The details are in the design wiki, as the implementation happens more discussions can happen here.
> https://cwiki.apache.org/confluence/display/SQOOP/Delta+Fetch+And+Merge+Design#DeltaFetchAndMergeDesign-Howtogetoutputfromconnectortosqoop?
> The goal is to dynamically inject a IncrementalConfig instance into the FromJobConfiguration. The current MFromConfig and MToConfig can already hold a list of configs, and a strong sentiment was expressed to keep it as a list, why not for the first time actually make use of it and group the incremental related configs in one config object
> This task will prepare the FromJobConfiguration from the job config data, ExtractorContext with the relevant values from the prev job run 
> This task will prepare the ToJobConfiguration from the job config data, LoaderContext with the relevant values from the prev job run if any
> We will use DistributedCache to get State information from the Extractor and Loader out and finally persist it into the sqoop repository depending on SQOOP-1804 once the outputcommitter commit is called



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)