[jira] [Assigned] (SQOOP-2579) Sqoop2: Refactore RepositoryManager to not load structure from repository when loading jobs and links

["Abraham Fine (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SQOOP-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abraham Fine reassigned SQOOP-2579:
-----------------------------------

    Assignee: Abraham Fine

> Sqoop2: Refactore RepositoryManager to not load structure from repository when loading jobs and links
> -----------------------------------------------------------------------------------------------------
>
>                 Key: SQOOP-2579
>                 URL: https://issues.apache.org/jira/browse/SQOOP-2579
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.99.6
>            Reporter: Jarek Jarcec Cecho
>            Assignee: Abraham Fine
>
> This one will take a bit longer to explain, so please _beer_ with me :)
> The current flow of configuration class is this one:
> * Connector developer defines class and marks inputs that are required from users with [{{@Input}}|https://github.com/apache/sqoop/blob/sqoop2/common/src/main/java/org/apache/sqoop/model/Input.java] annotation. This annotation can have several parameters ({{sensitive}}, {{size}}, ...).
> * As we can't use those classes everywhere (mainly we can't send them to clients over the wire nor serialize them to repository), we have [{{ConfigUtil}}|https://github.com/apache/sqoop/blob/sqoop2/common/src/main/java/org/apache/sqoop/model/ConfigUtils.java] class that serializes them to [model objects|https://github.com/apache/sqoop/tree/sqoop2/common/src/main/java/org/apache/sqoop/model].
> * Now {{ConnectorManager}} takes the model objects and persist them in memory for the lifetime of Sqoop 2 server process (after restart we simply run the {{ConfigUtils}} again, so nothing is lost).
> * The {{RepositoryManager}} takes the model objects and persist them in database. This is desirable because this way we can detect if the structures actually changed (e.g. connector code has been updated) and we can call upgrade routines.
> On REST layer, we have following behavior:
> * For {{/connector/}} we always call [{{ConnectorManager}}|https://github.com/apache/sqoop/blob/sqoop2/server/src/main/java/org/apache/sqoop/handler/ConnectorRequestHandler.java] to get the in memory structures.
> * For {{/job}} and {{/links}} - e.g. to get actual values for those model objects - we are using [{{RepositoryManager}}|https://github.com/apache/sqoop/blob/sqoop2/server/src/main/java/org/apache/sqoop/handler/JobRequestHandler.java] instead. As the {{RepositoryManager}} is currently not using {{ConnectorManager}} it uses the model objects that are serialized in the database instead.
> This behavior is a bit limiting because now if we want to improve the model classes and add new information ([Validations|https://issues.apache.org/jira/browse/SQOOP-1442], [Sensitive keys for maps|https://issues.apache.org/jira/browse/SQOOP-2549]) we absolutely have to serialize that information in database. And that means that we have to do database upgrade, that means that if this information is changed we have to do data upgrade, ... Even though that this is not required as those kind of things can be kept in memory as we don't need to do any action when they change.
> Hence I would like to propose changing current behavior and making {{RepositoryManager}} use {{ConnectroManager}} to read jobs and links objects. This way we can have more in memory state and what state is persisted in database.
> Please note that I'm not suggesting to drop the model classes from repository - we still need them to detect if metadata (the {{@Input}} annotation) has changed and run upgrade if required. I'm only suggesting to not used this serialized metadata if we are reading job and links objects.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)