You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/05/17 00:42:00 UTC

[jira] [Work logged] (GOBBLIN-1150) change spec store table to use json type

     [ https://issues.apache.org/jira/browse/GOBBLIN-1150?focusedWorklogId=434125&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434125 ]

ASF GitHub Bot logged work on GOBBLIN-1150:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 17/May/20 00:41
            Start Date: 17/May/20 00:41
    Worklog Time Spent: 10m 
      Work Description: sv2000 commented on a change in pull request #2988:
URL: https://github.com/apache/incubator-gobblin/pull/2988#discussion_r426203594



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_store/MysqlSpecStore.java
##########
@@ -65,31 +65,49 @@
 
   private static final String CREATE_TABLE_STATEMENT =
       "CREATE TABLE IF NOT EXISTS %s (spec_uri VARCHAR(128) NOT NULL, tag VARCHAR(128) NOT NULL, spec LONGBLOB, PRIMARY KEY (spec_uri))";
+  private static final String CREATE_TABLE_STATEMENT_V2 =

Review comment:
       Isn't this schema backwards compatible with the V1 schema i.e. you are simply adding new columns? We can retain the blob format for the "spec" column as before and define a new column "specV2" which stores the spec in JSON format. If so, do we need to support two different versions of the table?

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_store/MysqlSpecStore.java
##########
@@ -65,31 +65,49 @@
 
   private static final String CREATE_TABLE_STATEMENT =
       "CREATE TABLE IF NOT EXISTS %s (spec_uri VARCHAR(128) NOT NULL, tag VARCHAR(128) NOT NULL, spec LONGBLOB, PRIMARY KEY (spec_uri))";
+  private static final String CREATE_TABLE_STATEMENT_V2 =
+      "CREATE TABLE IF NOT EXISTS %s (spec_uri VARCHAR(128) NOT NULL, flow_group VARCHAR(128), flow_name VARCHAR(128), "

Review comment:
       I think the schema can be enhanced with a few additional fields such as requesterId, a boolean field "isRunImmediately" to indicate if the scheduled flow is also a run immediately flow, a futuristic field "owningGroup" for flows which are owned by a group, "timezone" to indicate the timezone the cron schedule applies to. Also, does user_to_proxy need to be a top level field? It applies only when the orchestrator is Azkaban and can probably be queried from the JSON spec.

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_store/MysqlSpecStore.java
##########
@@ -140,11 +172,22 @@ public boolean deleteSpec(Spec spec) throws IOException {
   @Override
   public boolean deleteSpec(URI specUri) throws IOException {
     try (Connection connection = this.dataSource.getConnection();
-        PreparedStatement statement = connection.prepareStatement(String.format(DELETE_STATEMENT, this.tableName))) {
+        PreparedStatement statement = connection.prepareStatement(String.format(DELETE_STATEMENT, this.tableName));
+        PreparedStatement statementV2 = connection.prepareStatement(String.format(DELETE_STATEMENT, this.tableNameV2))) {
+
       statement.setString(1, specUri.toString());
-      int result = statement.executeUpdate();
+      statementV2.setString(1, specUri.toString());
+
+      int resultV2 = statementV2.executeUpdate();
+      if (this.writeToOldTable) {
+        int result = statement.executeUpdate();
+        if (resultV2 != result) {

Review comment:
       Should we throw an exception and fail the deleteSpec, instead of logging and moving on?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 434125)
    Remaining Estimate: 0h
            Time Spent: 10m

> change spec store table to use json type
> ----------------------------------------
>
>                 Key: GOBBLIN-1150
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1150
>             Project: Apache Gobblin
>          Issue Type: Improvement
>            Reporter: Arjun Singh Bora
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)