You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "lightzhao (via GitHub)" <gi...@apache.org> on 2023/06/26 10:27:52 UTC

[GitHub] [seatunnel] lightzhao opened a new pull request, #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

lightzhao opened a new pull request, #4975:
URL: https://github.com/apache/seatunnel/pull/4975

   <!--
   
   Thank you for contributing to SeaTunnel! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   ## Contribution Checklist
   
     - Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
   
     - Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
   
     - Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
   
   -->
   
   ## Purpose of this pull request
   
   <!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
   Support redis to set expiration time.
   ## Check list
   
   * [ ] Code changed are covered with tests, or it does not need tests for reason:
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2)
     2. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     3. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
   * [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] TyrantLucifer commented on a diff in pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer commented on code in PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#discussion_r1244615159


##########
seatunnel-connectors-v2/connector-redis/src/main/java/org/apache/seatunnel/connectors/seatunnel/redis/config/RedisConfig.java:
##########
@@ -102,6 +102,12 @@ public enum HashKeyParseMode {
                     .withDescription(
                             "hash key parse mode, support all or kv, default value is all");
 
+    public static final Option<Long> EXPIRE =
+            Options.key("expire")
+                    .longType()
+                    .noDefaultValue()

Review Comment:
   default value = 0



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] lightzhao commented on a diff in pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "lightzhao (via GitHub)" <gi...@apache.org>.
lightzhao commented on code in PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#discussion_r1244625471


##########
seatunnel-connectors-v2/connector-redis/src/main/java/org/apache/seatunnel/connectors/seatunnel/redis/config/RedisParameters.java:
##########
@@ -89,6 +90,9 @@ public void buildWithConfig(Config config) {
         if (config.hasPath(RedisConfig.KEY_PATTERN.key())) {
             this.keysPattern = config.getString(RedisConfig.KEY_PATTERN.key());
         }
+        if (config.hasPath(RedisConfig.EXPIRE.key())) {
+            this.expire = config.getLong(RedisConfig.EXPIRE.key());

Review Comment:
   Default values have been added. In Redis, the default value of the EXPIRE command is -1, and keys are not automatically expired by default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] liugddx commented on pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#issuecomment-1608808064

   Does the parameter make sense?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] lightzhao commented on pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "lightzhao (via GitHub)" <gi...@apache.org>.
lightzhao commented on PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#issuecomment-1608847243

   > Does the parameter make sense?
   
   Surely it makes sense, if a key is created but no expiration is set for it, then the key will remain in memory forever. The production environment needs to set the expiration time, otherwise the memory will be greatly wasted.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] lightzhao commented on pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "lightzhao (via GitHub)" <gi...@apache.org>.
lightzhao commented on PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#issuecomment-1614020950

   @EricJoy2048 @TyrantLucifer @liugddx PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] EricJoy2048 commented on pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#issuecomment-1622960173

   Can you add e2e test for this pr?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] lightzhao commented on a diff in pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "lightzhao (via GitHub)" <gi...@apache.org>.
lightzhao commented on code in PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#discussion_r1267896639


##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-redis-e2e/src/test/java/org/apache/seatunnel/e2e/connector/redis/RedisIT.java:
##########
@@ -192,4 +192,15 @@ public void testRedis(TestContainer container) throws IOException, InterruptedEx
         jedis.del("key_list");
         Assertions.assertEquals(0, jedis.llen("key_list"));
     }
+
+    @TestTemplate
+    public void testRedisWithExpire(TestContainer container)
+            throws IOException, InterruptedException {
+        Container.ExecResult execResult = container.executeJob("/redis-to-redis-1.conf");

Review Comment:
   done, PTAL.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] liugddx commented on pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#issuecomment-1641859069

   Wait for ci to complete


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] liugddx commented on pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#issuecomment-1643354192

   @TyrantLucifer 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] ic4y commented on pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "ic4y (via GitHub)" <gi...@apache.org>.
ic4y commented on PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#issuecomment-1666453989

   @TyrantLucifer


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] lightzhao commented on pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "lightzhao (via GitHub)" <gi...@apache.org>.
lightzhao commented on PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#issuecomment-1659810879

   @liugddx @TaoZex PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] lightzhao commented on pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "lightzhao (via GitHub)" <gi...@apache.org>.
lightzhao commented on PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#issuecomment-1661859402

   @EricJoy2048 @TyrantLucifer @liugddx @TaoZex @Hisoka-X PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] lightzhao commented on pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "lightzhao (via GitHub)" <gi...@apache.org>.
lightzhao commented on PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#issuecomment-1637449283

   > Can you add e2e test for this pr?
   
   done.PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] EricJoy2048 commented on a diff in pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on code in PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#discussion_r1244617032


##########
seatunnel-connectors-v2/connector-redis/src/main/java/org/apache/seatunnel/connectors/seatunnel/redis/config/RedisParameters.java:
##########
@@ -89,6 +90,9 @@ public void buildWithConfig(Config config) {
         if (config.hasPath(RedisConfig.KEY_PATTERN.key())) {
             this.keysPattern = config.getString(RedisConfig.KEY_PATTERN.key());
         }
+        if (config.hasPath(RedisConfig.EXPIRE.key())) {
+            this.expire = config.getLong(RedisConfig.EXPIRE.key());

Review Comment:
   It is suggest to use `EXPIRE.defaultvalue()`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] liugddx merged pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx merged PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] TaoZex commented on a diff in pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "TaoZex (via GitHub)" <gi...@apache.org>.
TaoZex commented on code in PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#discussion_r1267504782


##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-redis-e2e/src/test/java/org/apache/seatunnel/e2e/connector/redis/RedisIT.java:
##########
@@ -192,4 +192,15 @@ public void testRedis(TestContainer container) throws IOException, InterruptedEx
         jedis.del("key_list");
         Assertions.assertEquals(0, jedis.llen("key_list"));
     }
+
+    @TestTemplate
+    public void testRedisWithExpire(TestContainer container)
+            throws IOException, InterruptedException {
+        Container.ExecResult execResult = container.executeJob("/redis-to-redis-1.conf");

Review Comment:
   redis-to-redis-1.conf file name is not clear.
   How about use redis-to-redis-expire.conf?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] EricJoy2048 commented on pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#issuecomment-1672607700

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] liugddx commented on a diff in pull request #4975: [Feature][Connector-v2][RedisSink]Support redis to set expiration time.

Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on code in PR #4975:
URL: https://github.com/apache/seatunnel/pull/4975#discussion_r1264929494


##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-redis-e2e/src/test/java/org/apache/seatunnel/e2e/connector/redis/RedisIT.java:
##########
@@ -192,4 +192,15 @@ public void testRedis(TestContainer container) throws IOException, InterruptedEx
         jedis.del("key_list");
         Assertions.assertEquals(0, jedis.llen("key_list"));
     }
+
+    @TestTemplate
+    public void testRedisWithExpire(TestContainer container)
+            throws IOException, InterruptedException {
+        Container.ExecResult execResult = container.executeJob("/redis-to-redis-1.conf");
+        Assertions.assertEquals(0, execResult.getExitCode());
+        Assertions.assertEquals(100, jedis.llen("key_list"));
+        // Clear data to prevent data duplication in the next TestContainer
+        jedis.del("key_list");

Review Comment:
   Use sleep instead of del to test this feature?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org