You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "zhangshenghang (via GitHub)" <gi...@apache.org> on 2024/07/09 13:51:12 UTC

[PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

zhangshenghang opened a new pull request, #7148:
URL: https://github.com/apache/seatunnel/pull/7148

   ### Purpose of this pull request
   
   - Add Hbase Source parameter to support large data volume reading and optimize performance
     - Support caching parameter settings
     - Support batch parameter settings
     - Support cache_blocks parameter settings
   - Delete unused code
   
   ### How was this patch tested?
   
   Running locally and on servers
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [x] If you are contributing the connector code, please check that the following files are updated:
     1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2)
     2. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     3. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
   * [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "zhilinli123 (via GitHub)" <gi...@apache.org>.
zhilinli123 commented on PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#issuecomment-2222725381

   hbase source & sink The document is changed to v2, Add Chinese document
   
   reference :
   https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Mysql.md
   https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/source/Mysql.md
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "TaoZex (via GitHub)" <gi...@apache.org>.
TaoZex merged PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on code in PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#discussion_r1672307129


##########
seatunnel-connectors-v2/connector-hbase/src/main/java/org/apache/seatunnel/connectors/seatunnel/hbase/source/HbaseSourceReader.java:
##########
@@ -39,22 +40,21 @@
 import java.io.IOException;
 import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.Deque;
-import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
-import java.util.Set;
 import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.stream.Collectors;
 
 @Slf4j
 public class HbaseSourceReader implements SourceReader<SeaTunnelRow, HbaseSourceSplit> {
-    private static final String ROW_KEY = "rowkey";
+    private final String ROW_KEY = "rowkey";

Review Comment:
   > Why remove static?
   
   The previous methods using ROW_KEY were all non-static methods. Using this.ROW_KEY to call private final static String ROW_KEY = "rowkey"; is not a good choice.
   
   Here is how to use ROW_KEY in historical code:
   ![image](https://github.com/apache/seatunnel/assets/29418975/e56abcb6-be10-4ff4-ad15-91624b93fe7b)
   ![image](https://github.com/apache/seatunnel/assets/29418975/80b6060f-db4f-407f-83ce-5c11b0c32d03)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#issuecomment-2238631051

   > Please add test case in
   > 
   > https://github.com/apache/seatunnel/blob/2058f5d9c31af97c39bfbad52e2e28cdaffa7892/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-hbase-e2e/src/test/java/org/apache/seatunnel/e2e/connector/hbase/HbaseIT.java#L56
   
   Okay, I will add it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#discussion_r1673581542


##########
seatunnel-connectors-v2/connector-hbase/src/main/java/org/apache/seatunnel/connectors/seatunnel/hbase/source/HbaseSourceReader.java:
##########
@@ -39,22 +40,21 @@
 import java.io.IOException;
 import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.Deque;
-import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
-import java.util.Set;
 import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.stream.Collectors;
 
 @Slf4j
 public class HbaseSourceReader implements SourceReader<SeaTunnelRow, HbaseSourceSplit> {
-    private static final String ROW_KEY = "rowkey";
+    private final String ROW_KEY = "rowkey";

Review Comment:
   how about remove `this` when call `ROW_KEY`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#discussion_r1673584844


##########
docs/en/connector-v2/source/Hbase.md:
##########
@@ -17,75 +17,80 @@ Read data from Apache Hbase.
 
 ## Options
 
-|        name        |  type  | required | default value |
-|--------------------|--------|----------|---------------|
-| zookeeper_quorum   | string | yes      | -             |
-| table              | string | yes      | -             |
-| query_columns      | list   | yes      | -             |
-| schema             | config | yes      | -             |
-| hbase_extra_config | string | no       | -             |
-| common-options     |        | no       | -             |
+|        Name        |  Type   | Required | Default |
+|--------------------|---------|----------|---------|
+| zookeeper_quorum   | string  | Yes      | -       |
+| table              | string  | Yes      | -       |
+| schema             | config  | Yes      | -       |
+| hbase_extra_config | string  | No       | -       |
+| caching            | int     | No       | -1      |
+| batch              | int     | No       | -1      |
+| cache_blocks       | boolean | No       | false   |
+| common-options     |         | No       | -       |
 
 ### zookeeper_quorum [string]
 
-The zookeeper cluster host of hbase, example: "hadoop001:2181,hadoop002:2181,hadoop003:2181"
+The zookeeper quorum for Hbase cluster hosts, e.g., "hadoop001:2181,hadoop002:2181,hadoop003:2181".
 
 ### table [string]
 
-The table name you want to write, example: "seatunnel"
-
-### query_columns [list]
-
-The column name which you want to query in the table. If you want to query the rowkey column, please set "rowkey" in query_columns.
-Other column format should be: columnFamily:columnName, example: ["rowkey", "columnFamily1:column1", "columnFamily1:column1", "columnFamily2:column1"]

Review Comment:
   +1 from my side.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#issuecomment-2259608596

   > Please add test case in
   > 
   > https://github.com/apache/seatunnel/blob/2058f5d9c31af97c39bfbad52e2e28cdaffa7892/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-hbase-e2e/src/test/java/org/apache/seatunnel/e2e/connector/hbase/HbaseIT.java#L56
   
   @Hisoka-X add completed , Please confirm if there is any problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#issuecomment-2222392578

   > hi, can you add the hbase sink document together
   Hi, the HBase sink documentation is available now. What do you want to do specifically?
   ![image](https://github.com/apache/seatunnel/assets/29418975/288663ae-05c2-41cb-80e4-c7d9f59028bf)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#discussion_r1673583380


##########
docs/en/connector-v2/source/Hbase.md:
##########
@@ -17,75 +17,80 @@ Read data from Apache Hbase.
 
 ## Options
 
-|        name        |  type  | required | default value |
-|--------------------|--------|----------|---------------|
-| zookeeper_quorum   | string | yes      | -             |
-| table              | string | yes      | -             |
-| query_columns      | list   | yes      | -             |
-| schema             | config | yes      | -             |
-| hbase_extra_config | string | no       | -             |
-| common-options     |        | no       | -             |
+|        Name        |  Type   | Required | Default |
+|--------------------|---------|----------|---------|
+| zookeeper_quorum   | string  | Yes      | -       |
+| table              | string  | Yes      | -       |
+| schema             | config  | Yes      | -       |
+| hbase_extra_config | string  | No       | -       |
+| caching            | int     | No       | -1      |
+| batch              | int     | No       | -1      |
+| cache_blocks       | boolean | No       | false   |
+| common-options     |         | No       | -       |
 
 ### zookeeper_quorum [string]
 
-The zookeeper cluster host of hbase, example: "hadoop001:2181,hadoop002:2181,hadoop003:2181"
+The zookeeper quorum for Hbase cluster hosts, e.g., "hadoop001:2181,hadoop002:2181,hadoop003:2181".
 
 ### table [string]
 
-The table name you want to write, example: "seatunnel"
-
-### query_columns [list]
-
-The column name which you want to query in the table. If you want to query the rowkey column, please set "rowkey" in query_columns.
-Other column format should be: columnFamily:columnName, example: ["rowkey", "columnFamily1:column1", "columnFamily1:column1", "columnFamily2:column1"]

Review Comment:
   cc @EricJoy2048 @TyrantLucifer @TaoZex @hailin0 from #6348



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on code in PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#discussion_r1672294166


##########
docs/en/connector-v2/source/Hbase.md:
##########
@@ -17,75 +17,80 @@ Read data from Apache Hbase.
 
 ## Options
 
-|        name        |  type  | required | default value |
-|--------------------|--------|----------|---------------|
-| zookeeper_quorum   | string | yes      | -             |
-| table              | string | yes      | -             |
-| query_columns      | list   | yes      | -             |
-| schema             | config | yes      | -             |
-| hbase_extra_config | string | no       | -             |
-| common-options     |        | no       | -             |
+|        Name        |  Type   | Required | Default |
+|--------------------|---------|----------|---------|
+| zookeeper_quorum   | string  | Yes      | -       |
+| table              | string  | Yes      | -       |
+| schema             | config  | Yes      | -       |
+| hbase_extra_config | string  | No       | -       |
+| caching            | int     | No       | -1      |
+| batch              | int     | No       | -1      |
+| cache_blocks       | boolean | No       | false   |
+| common-options     |         | No       | -       |
 
 ### zookeeper_quorum [string]
 
-The zookeeper cluster host of hbase, example: "hadoop001:2181,hadoop002:2181,hadoop003:2181"
+The zookeeper quorum for Hbase cluster hosts, e.g., "hadoop001:2181,hadoop002:2181,hadoop003:2181".
 
 ### table [string]
 
-The table name you want to write, example: "seatunnel"
-
-### query_columns [list]
-
-The column name which you want to query in the table. If you want to query the rowkey column, please set "rowkey" in query_columns.
-Other column format should be: columnFamily:columnName, example: ["rowkey", "columnFamily1:column1", "columnFamily1:column1", "columnFamily2:column1"]

Review Comment:
   > Why remove this feature?
   
   This feature was not implemented in previous versions, and query_columns was an invalid configuration. After this modification, the query columns use the schema configuration. It is redundant to set both schema and query_columns at the same time.
   
   The columns configured by the schema are also the columns for querying
   ```
   schema = {
               columns = [
                     {
                        name = rowkey
                        type = string
                     },
                     {
                        name = "info1:age"
                        type = int
                     },
                     {
                         name = "info1:name"
                         type = string
                     }
   ]
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on code in PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#discussion_r1672294166


##########
docs/en/connector-v2/source/Hbase.md:
##########
@@ -17,75 +17,80 @@ Read data from Apache Hbase.
 
 ## Options
 
-|        name        |  type  | required | default value |
-|--------------------|--------|----------|---------------|
-| zookeeper_quorum   | string | yes      | -             |
-| table              | string | yes      | -             |
-| query_columns      | list   | yes      | -             |
-| schema             | config | yes      | -             |
-| hbase_extra_config | string | no       | -             |
-| common-options     |        | no       | -             |
+|        Name        |  Type   | Required | Default |
+|--------------------|---------|----------|---------|
+| zookeeper_quorum   | string  | Yes      | -       |
+| table              | string  | Yes      | -       |
+| schema             | config  | Yes      | -       |
+| hbase_extra_config | string  | No       | -       |
+| caching            | int     | No       | -1      |
+| batch              | int     | No       | -1      |
+| cache_blocks       | boolean | No       | false   |
+| common-options     |         | No       | -       |
 
 ### zookeeper_quorum [string]
 
-The zookeeper cluster host of hbase, example: "hadoop001:2181,hadoop002:2181,hadoop003:2181"
+The zookeeper quorum for Hbase cluster hosts, e.g., "hadoop001:2181,hadoop002:2181,hadoop003:2181".
 
 ### table [string]
 
-The table name you want to write, example: "seatunnel"
-
-### query_columns [list]
-
-The column name which you want to query in the table. If you want to query the rowkey column, please set "rowkey" in query_columns.
-Other column format should be: columnFamily:columnName, example: ["rowkey", "columnFamily1:column1", "columnFamily1:column1", "columnFamily2:column1"]

Review Comment:
   > Why remove this feature?
   
   This feature was not implemented in previous versions, and query_columns was an invalid configuration. After this modification, the query columns use the schema configuration. It is redundant to set both schema and query_columns at the same time.
   
   The columns configured in the schema are equivalent to the columns in the query
   ```
   schema = {
               columns = [
                     {
                        name = rowkey
                        type = string
                     },
                     {
                        name = "info1:age"
                        type = int
                     },
                     {
                         name = "info1:name"
                         type = string
                     }
   ]
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "zhilinli123 (via GitHub)" <gi...@apache.org>.
zhilinli123 commented on PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#issuecomment-2222352190

   hi, can you add the hbase sink document together
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#issuecomment-2230060784

   > > > Cache is often used for frequently querying hot data. In data synchronization read scenarios, data is usually queried only once. Is it effective to configure cache-related settings in this case? Looking forward to your reply.
   > > 
   > > 
   > > You are right. In the case of synchronous data, it only needs to be queried once, so I set the default value of `cache_blocks` to false. The default cache_blocks value of Hbase Scan is True. At the same time, I am not sure whether any user will use cache_blocks = true, so I set the cache_blocks parameter to an optional parameter for special cases.
   > > ```
   > > public static final Option<Boolean> HBASE_CACHE_BLOCKS_CONFIG =
   > > Options.key("cache_blocks")
   > > .booleanType()
   > > .defaultValue(false)
   > > .withDescription(
   > > "When it is false, data blocks are not cached. When it is true, data blocks are cached. This value should be set to false when scanning a large amount of data to reduce memory consumption. The default value is false");
   > > ```
   > 
   > Thank you for your reply. I believe setting the batch size is necessary. Should we keep the cache enabled? I would like to hear your suggestions. @Hisoka-X @hailin0
   
   OK for me. Give the choice to users.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#issuecomment-2238515154

   Please add test case in https://github.com/apache/seatunnel/blob/2058f5d9c31af97c39bfbad52e2e28cdaffa7892/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-hbase-e2e/src/test/java/org/apache/seatunnel/e2e/connector/hbase/HbaseIT.java#L56


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#discussion_r1672178021


##########
seatunnel-connectors-v2/connector-hbase/src/main/java/org/apache/seatunnel/connectors/seatunnel/hbase/source/HbaseSourceReader.java:
##########
@@ -39,22 +40,21 @@
 import java.io.IOException;
 import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.Deque;
-import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
-import java.util.Set;
 import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.stream.Collectors;
 
 @Slf4j
 public class HbaseSourceReader implements SourceReader<SeaTunnelRow, HbaseSourceSplit> {
-    private static final String ROW_KEY = "rowkey";
+    private final String ROW_KEY = "rowkey";

Review Comment:
   Why remove static?



##########
docs/en/connector-v2/source/Hbase.md:
##########
@@ -17,75 +17,80 @@ Read data from Apache Hbase.
 
 ## Options
 
-|        name        |  type  | required | default value |
-|--------------------|--------|----------|---------------|
-| zookeeper_quorum   | string | yes      | -             |
-| table              | string | yes      | -             |
-| query_columns      | list   | yes      | -             |
-| schema             | config | yes      | -             |
-| hbase_extra_config | string | no       | -             |
-| common-options     |        | no       | -             |
+|        Name        |  Type   | Required | Default |
+|--------------------|---------|----------|---------|
+| zookeeper_quorum   | string  | Yes      | -       |
+| table              | string  | Yes      | -       |
+| schema             | config  | Yes      | -       |
+| hbase_extra_config | string  | No       | -       |
+| caching            | int     | No       | -1      |
+| batch              | int     | No       | -1      |
+| cache_blocks       | boolean | No       | false   |
+| common-options     |         | No       | -       |
 
 ### zookeeper_quorum [string]
 
-The zookeeper cluster host of hbase, example: "hadoop001:2181,hadoop002:2181,hadoop003:2181"
+The zookeeper quorum for Hbase cluster hosts, e.g., "hadoop001:2181,hadoop002:2181,hadoop003:2181".
 
 ### table [string]
 
-The table name you want to write, example: "seatunnel"
-
-### query_columns [list]
-
-The column name which you want to query in the table. If you want to query the rowkey column, please set "rowkey" in query_columns.
-Other column format should be: columnFamily:columnName, example: ["rowkey", "columnFamily1:column1", "columnFamily1:column1", "columnFamily2:column1"]

Review Comment:
   Why remove this feature?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on code in PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#discussion_r1673596375


##########
seatunnel-connectors-v2/connector-hbase/src/main/java/org/apache/seatunnel/connectors/seatunnel/hbase/source/HbaseSourceReader.java:
##########
@@ -39,22 +40,21 @@
 import java.io.IOException;
 import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.Deque;
-import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
-import java.util.Set;
 import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.stream.Collectors;
 
 @Slf4j
 public class HbaseSourceReader implements SourceReader<SeaTunnelRow, HbaseSourceSplit> {
-    private static final String ROW_KEY = "rowkey";
+    private final String ROW_KEY = "rowkey";

Review Comment:
   > how about remove `this` when call `ROW_KEY`?
   
   Thank you for your reminder, You are right, this will be better, I will optimize it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#issuecomment-2227768820

   > Cache is often used for frequently querying hot data. In data synchronization read scenarios, data is usually queried only once. Is it effective to configure cache-related settings in this case? Looking forward to your reply.
   
   You are right. In the case of synchronous data, it only needs to be queried once, so I set the default value of `cache_blocks` to false. The default cache_blocks value of Hbase Scan is True. At the same time, I am not sure whether any user will use cache_blocks = true, so I set the cache_blocks parameter to an optional parameter for special cases.
   
   ```
   public static final Option<Boolean> HBASE_CACHE_BLOCKS_CONFIG =
   Options.key("cache_blocks")
   .booleanType()
   .defaultValue(false)
   .withDescription(
   "When it is false, data blocks are not cached. When it is true, data blocks are cached. This value should be set to false when scanning a large amount of data to reduce memory consumption. The default value is false");
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][connector-v2-hbase]fix and optimize hbase source problem [seatunnel]

Posted by "TaoZex (via GitHub)" <gi...@apache.org>.
TaoZex commented on PR #7148:
URL: https://github.com/apache/seatunnel/pull/7148#issuecomment-2230044632

   > > Cache is often used for frequently querying hot data. In data synchronization read scenarios, data is usually queried only once. Is it effective to configure cache-related settings in this case? Looking forward to your reply.
   > 
   > You are right. In the case of synchronous data, it only needs to be queried once, so I set the default value of `cache_blocks` to false. The default cache_blocks value of Hbase Scan is True. At the same time, I am not sure whether any user will use cache_blocks = true, so I set the cache_blocks parameter to an optional parameter for special cases.
   > 
   > ```
   > public static final Option<Boolean> HBASE_CACHE_BLOCKS_CONFIG =
   > Options.key("cache_blocks")
   > .booleanType()
   > .defaultValue(false)
   > .withDescription(
   > "When it is false, data blocks are not cached. When it is true, data blocks are cached. This value should be set to false when scanning a large amount of data to reduce memory consumption. The default value is false");
   > ```
   
   Thank you for your reply. I believe setting the batch size is necessary. Should we keep the cache enabled? I would like to hear your suggestions. @Hisoka-X @hailin0 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org