You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by GitBox <gi...@apache.org> on 2022/10/31 20:30:30 UTC

[GitHub] [gobblin] ZihanLi58 opened a new pull request, #3595: [GOBBLIN-1737] Fix bug when using mysql user quota manager

ZihanLi58 opened a new pull request, #3595:
URL: https://github.com/apache/gobblin/pull/3595

   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
   
   
   ### JIRA
   - [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
       - https://issues.apache.org/jira/browse/GOBBLIN-1737
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if applicable):
   Change the way we get config to to consistent with other mysql store
   Only check quota in compiler when the flow compile successful and explain flag is not set
   Change the way we initialize the quota Manager.
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   Deploy locally and verify the mysql quota manager works as expected
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
       1. Subject is separated from body by a blank line
       2. Subject is limited to 50 characters
       3. Subject does not end with a period
       4. Subject uses the imperative mood ("add", not "adding")
       5. Body wraps at 72 characters
       6. Body explains "what" and "why", not "how"
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] umustafi commented on a diff in pull request #3595: [GOBBLIN-1737] Fix bug when using mysql user quota manager

Posted by GitBox <gi...@apache.org>.
umustafi commented on code in PR #3595:
URL: https://github.com/apache/gobblin/pull/3595#discussion_r1014521337


##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManager.java:
##########
@@ -390,7 +390,8 @@ public synchronized void setActive(boolean active) {
 
        this.dagManagerMetrics.activate();
 
-        UserQuotaManager quotaManager = new InMemoryUserQuotaManager(config);
+        UserQuotaManager quotaManager = GobblinConstructorUtils.invokeConstructor(UserQuotaManager.class,
+            ConfigUtils.getString(config, ServiceConfigKeys.QUOTA_MANAGER_CLASS, ServiceConfigKeys.DEFAULT_QUOTA_MANAGER), config);

Review Comment:
   do you see other types of quota manager besides in memory actually being used? 



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/MysqlUserQuotaManager.java:
##########
@@ -295,7 +302,8 @@ public MysqlQuotaStore(BasicDataSource dataSource, String tableName)
       DECREASE_FLOWGROUP_COUNT_SQL = "UPDATE " + tableName + " SET flowgroup_count=flowgroup_count-1 WHERE name = ?";
       DELETE_USER_SQL = "DELETE FROM " + tableName + " WHERE name = ? AND user_count<1 AND flowgroup_count<1";
 
-      String createQuotaTable = "CREATE TABLE IF NOT EXISTS " + tableName + " (name VARCHAR(20) CHARACTER SET latin1 NOT NULL, "
+      //Increase the length of name as we include the executor uri in it
+      String createQuotaTable = "CREATE TABLE IF NOT EXISTS " + tableName + " (name VARCHAR(500) CHARACTER SET latin1 NOT NULL, "

Review Comment:
   anything else contained in table name? how long is executor uri max? It will be helpful to give idea of what table name will be 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] ZihanLi58 commented on a diff in pull request #3595: [GOBBLIN-1737] Fix bug when using mysql user quota manager

Posted by GitBox <gi...@apache.org>.
ZihanLi58 commented on code in PR #3595:
URL: https://github.com/apache/gobblin/pull/3595#discussion_r1014530751


##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/MysqlUserQuotaManager.java:
##########
@@ -295,7 +302,8 @@ public MysqlQuotaStore(BasicDataSource dataSource, String tableName)
       DECREASE_FLOWGROUP_COUNT_SQL = "UPDATE " + tableName + " SET flowgroup_count=flowgroup_count-1 WHERE name = ?";
       DELETE_USER_SQL = "DELETE FROM " + tableName + " WHERE name = ? AND user_count<1 AND flowgroup_count<1";
 
-      String createQuotaTable = "CREATE TABLE IF NOT EXISTS " + tableName + " (name VARCHAR(20) CHARACTER SET latin1 NOT NULL, "
+      //Increase the length of name as we include the executor uri in it
+      String createQuotaTable = "CREATE TABLE IF NOT EXISTS " + tableName + " (name VARCHAR(500) CHARACTER SET latin1 NOT NULL, "

Review Comment:
   Under name, it will be "user name for the quota" + "spec uri". I tried 200 and see it does not work so directly give 500



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] ZihanLi58 commented on a diff in pull request #3595: [GOBBLIN-1737] Fix bug when using mysql user quota manager

Posted by GitBox <gi...@apache.org>.
ZihanLi58 commented on code in PR #3595:
URL: https://github.com/apache/gobblin/pull/3595#discussion_r1014530369


##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManager.java:
##########
@@ -390,7 +390,8 @@ public synchronized void setActive(boolean active) {
 
        this.dagManagerMetrics.activate();
 
-        UserQuotaManager quotaManager = new InMemoryUserQuotaManager(config);
+        UserQuotaManager quotaManager = GobblinConstructorUtils.invokeConstructor(UserQuotaManager.class,
+            ConfigUtils.getString(config, ServiceConfigKeys.QUOTA_MANAGER_CLASS, ServiceConfigKeys.DEFAULT_QUOTA_MANAGER), config);

Review Comment:
   In warm standby mode, we need use mysql based quota manager



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] ZihanLi58 merged pull request #3595: [GOBBLIN-1737] Fix bug when using mysql user quota manager

Posted by GitBox <gi...@apache.org>.
ZihanLi58 merged PR #3595:
URL: https://github.com/apache/gobblin/pull/3595


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [gobblin] codecov-commenter commented on pull request #3595: [GOBBLIN-1737] Fix bug when using mysql user quota manager

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #3595:
URL: https://github.com/apache/gobblin/pull/3595#issuecomment-1297666762

   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3595?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3595](https://codecov.io/gh/apache/gobblin/pull/3595?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b4d8408) into [master](https://codecov.io/gh/apache/gobblin/commit/4d991a9bd872a98bc52dda2ec366ad7d77847f9a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4d991a9) will **decrease** coverage by `3.08%`.
   > The diff coverage is `n/a`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3595      +/-   ##
   ============================================
   - Coverage     46.88%   43.80%   -3.09%     
   + Complexity    10672     2060    -8612     
   ============================================
     Files          2120      408    -1712     
     Lines         83064    17618   -65446     
     Branches       9250     2153    -7097     
   ============================================
   - Hits          38942     7717   -31225     
   + Misses        40554     9042   -31512     
   + Partials       3568      859    -2709     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3595?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ervice/modules/core/GobblinServiceGuiceModule.java](https://codecov.io/gh/apache/gobblin/pull/3595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9jb3JlL0dvYmJsaW5TZXJ2aWNlR3VpY2VNb2R1bGUuamF2YQ==) | | |
   | [...ervice/modules/flow/BaseFlowToJobSpecCompiler.java](https://codecov.io/gh/apache/gobblin/pull/3595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93L0Jhc2VGbG93VG9Kb2JTcGVjQ29tcGlsZXIuamF2YQ==) | | |
   | [...blin/service/modules/orchestration/DagManager.java](https://codecov.io/gh/apache/gobblin/pull/3595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9vcmNoZXN0cmF0aW9uL0RhZ01hbmFnZXIuamF2YQ==) | | |
   | [...e/modules/orchestration/MysqlUserQuotaManager.java](https://codecov.io/gh/apache/gobblin/pull/3595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9vcmNoZXN0cmF0aW9uL015c3FsVXNlclF1b3RhTWFuYWdlci5qYXZh) | | |
   | [...he/gobblin/converter/avro/AvroToJsonConverter.java](https://codecov.io/gh/apache/gobblin/pull/3595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXZyby1qc29uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9hdnJvL0F2cm9Ub0pzb25Db252ZXJ0ZXIuamF2YQ==) | | |
   | [.../gobblin/source/DatePartitionedJsonFileSource.java](https://codecov.io/gh/apache/gobblin/pull/3595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9EYXRlUGFydGl0aW9uZWRKc29uRmlsZVNvdXJjZS5qYXZh) | | |
   | [...apache/gobblin/metrics/event/MultiTimingEvent.java](https://codecov.io/gh/apache/gobblin/pull/3595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9ldmVudC9NdWx0aVRpbWluZ0V2ZW50LmphdmE=) | | |
   | [...in/source/extractor/exception/SchemaException.java](https://codecov.io/gh/apache/gobblin/pull/3595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXhjZXB0aW9uL1NjaGVtYUV4Y2VwdGlvbi5qYXZh) | | |
   | [...bblin/compaction/mapreduce/orc/OrcValueMapper.java](https://codecov.io/gh/apache/gobblin/pull/3595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL29yYy9PcmNWYWx1ZU1hcHBlci5qYXZh) | | |
   | [...n/runtime/scheduler/ImmediateJobSpecScheduler.java](https://codecov.io/gh/apache/gobblin/pull/3595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc2NoZWR1bGVyL0ltbWVkaWF0ZUpvYlNwZWNTY2hlZHVsZXIuamF2YQ==) | | |
   | ... and [1702 more](https://codecov.io/gh/apache/gobblin/pull/3595/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org