You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/08/22 02:18:47 UTC

[GitHub] [doris] jacktengg opened a new pull request, #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model

jacktengg opened a new pull request, #11952:
URL: https://github.com/apache/doris/pull/11952

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   For debug purpose: add session variable agg_as_duplicate, when set to true, tables of aggregate key model and unique key model will be read as duplicate key model:
   ```
   mysql> create table t1(a int, b int) engine = olap unique key (a) distributed by hash(a) buckets 16 properties("replication_allocation" = "tag.location.default: 1");
   Query OK, 0 rows affected (0.28 sec)
   
   mysql> insert into t1 values (1,1),(2,1);
   Query OK, 2 rows affected (0.25 sec)
   {'label':'insert_e46de5e8df5246d4_ac3d12726252db7b', 'status':'VISIBLE', 'txnId':'2'}
   
   mysql> select * from t1;
   +------+------+
   | a    | b    |
   +------+------+
   |    2 |    1 |
   |    1 |    1 |
   +------+------+
   2 rows in set (0.47 sec)
   
   mysql> insert into t1 values (1,11),(2,11);
   Query OK, 2 rows affected (0.17 sec)
   {'label':'insert_7c44d5acd7b54bc7_82618edd8eef815b', 'status':'VISIBLE', 'txnId':'3'}
   
   mysql> select * from t1;
   +------+------+
   | a    | b    |
   +------+------+
   |    1 |   11 |
   |    2 |   11 |
   +------+------+
   2 rows in set (0.36 sec)
   
   mysql> show variables like "%agg%";
   +-----------------------------------+-------+
   | Variable_name                     | Value |
   +-----------------------------------+-------+
   | agg_as_duplicate                  | false |
   | disable_streaming_preaggregations | false |
   +-----------------------------------+-------+
   2 rows in set (0.00 sec)
   
   mysql> set agg_as_duplicate = true;
   Query OK, 0 rows affected (0.01 sec)
   
   mysql> select * from t1;
   +------+------+
   | a    | b    |
   +------+------+
   |    2 |    1 |
   |    2 |   11 |
   |    1 |    1 |
   |    1 |   11 |
   +------+------+
   4 rows in set (0.36 sec)
   ```
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
       - [ ] Yes
       - [ ] No
       - [ ] I don't know
   2. Has unit tests been added:
       - [ ] Yes
       - [ ] No
       - [ ] No Need
   3. Has document been added or modified:
       - [ ] Yes
       - [ ] No
       - [ ] No Need
   4. Does it need to update dependencies:
       - [ ] Yes
       - [ ] No
   5. Are there any changes that cannot be rolled back:
       - [ ] Yes (If Yes, please explain WHY)
       - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #11952:
URL: https://github.com/apache/doris/pull/11952#issuecomment-1245265868

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [doris] yiguolei merged pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model

Posted by GitBox <gi...@apache.org>.
yiguolei merged PR #11952:
URL: https://github.com/apache/doris/pull/11952


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [doris] morningman commented on a diff in pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model

Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #11952:
URL: https://github.com/apache/doris/pull/11952#discussion_r952040885


##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -525,6 +529,18 @@ public class SessionVariable implements Serializable, Writable {
     @VariableMgr.VarAttr(name = ENABLE_FUNCTION_PUSHDOWN)
     public boolean enableFunctionPushdown;
 
+    /**
+     * For debugg purpose, dont' merge unique key and agg key when reading data.
+     */
+    @VariableMgr.VarAttr(name = SKIP_STORAGE_ENGINE_MERGE)
+    private boolean skipStorageEngineMerge = false;
+
+    /**
+     * For debugg purpose, skip delte predicate when reading data.
+     */
+    @VariableMgr.VarAttr(name = SKIP_DELETE_PREDICATE)
+    private boolean skipDeletePredicate = false;

Review Comment:
   use `public`, or it may cause some serde probem.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [doris] yiguolei commented on a diff in pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model

Posted by GitBox <gi...@apache.org>.
yiguolei commented on code in PR #11952:
URL: https://github.com/apache/doris/pull/11952#discussion_r950966213


##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -519,6 +521,12 @@ public class SessionVariable implements Serializable, Writable {
     @VariableMgr.VarAttr(name = ENABLE_FUNCTION_PUSHDOWN)
     public boolean enableFunctionPushdown;
 
+    /**
+     * For debugg purpose, dont' merge unique key and agg key when reading data.
+     */
+    @VariableMgr.VarAttr(name = AGG_AS_DUPLICATE)
+    private boolean aggAsDuplicate = false;

Review Comment:
   the variable nameļ¼š skip_storage_engine_merge



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #11952:
URL: https://github.com/apache/doris/pull/11952#issuecomment-1221925001

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [doris] yiguolei commented on a diff in pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model

Posted by GitBox <gi...@apache.org>.
yiguolei commented on code in PR #11952:
URL: https://github.com/apache/doris/pull/11952#discussion_r950967191


##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -519,6 +521,12 @@ public class SessionVariable implements Serializable, Writable {
     @VariableMgr.VarAttr(name = ENABLE_FUNCTION_PUSHDOWN)
     public boolean enableFunctionPushdown;
 
+    /**
+     * For debugg purpose, dont' merge unique key and agg key when reading data.
+     */
+    @VariableMgr.VarAttr(name = AGG_AS_DUPLICATE)
+    private boolean aggAsDuplicate = false;
+
     public String getBlockEncryptionMode() {

Review Comment:
   also add another variable: skip_delete_predicate



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [doris] jacktengg commented on a diff in pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model

Posted by GitBox <gi...@apache.org>.
jacktengg commented on code in PR #11952:
URL: https://github.com/apache/doris/pull/11952#discussion_r950996455


##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -519,6 +521,12 @@ public class SessionVariable implements Serializable, Writable {
     @VariableMgr.VarAttr(name = ENABLE_FUNCTION_PUSHDOWN)
     public boolean enableFunctionPushdown;
 
+    /**
+     * For debugg purpose, dont' merge unique key and agg key when reading data.
+     */
+    @VariableMgr.VarAttr(name = AGG_AS_DUPLICATE)
+    private boolean aggAsDuplicate = false;

Review Comment:
   done



##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -519,6 +521,12 @@ public class SessionVariable implements Serializable, Writable {
     @VariableMgr.VarAttr(name = ENABLE_FUNCTION_PUSHDOWN)
     public boolean enableFunctionPushdown;
 
+    /**
+     * For debugg purpose, dont' merge unique key and agg key when reading data.
+     */
+    @VariableMgr.VarAttr(name = AGG_AS_DUPLICATE)
+    private boolean aggAsDuplicate = false;
+
     public String getBlockEncryptionMode() {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #11952:
URL: https://github.com/apache/doris/pull/11952#issuecomment-1221925037

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [doris] jacktengg commented on a diff in pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model

Posted by GitBox <gi...@apache.org>.
jacktengg commented on code in PR #11952:
URL: https://github.com/apache/doris/pull/11952#discussion_r966710970


##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -525,6 +529,18 @@ public class SessionVariable implements Serializable, Writable {
     @VariableMgr.VarAttr(name = ENABLE_FUNCTION_PUSHDOWN)
     public boolean enableFunctionPushdown;
 
+    /**
+     * For debugg purpose, dont' merge unique key and agg key when reading data.
+     */
+    @VariableMgr.VarAttr(name = SKIP_STORAGE_ENGINE_MERGE)
+    private boolean skipStorageEngineMerge = false;

Review Comment:
   updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [doris] yiguolei commented on a diff in pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model

Posted by GitBox <gi...@apache.org>.
yiguolei commented on code in PR #11952:
URL: https://github.com/apache/doris/pull/11952#discussion_r950967537


##########
be/src/vec/exec/volap_scanner.cpp:
##########
@@ -172,6 +172,11 @@ Status VOlapScanner::_init_tablet_reader_params(
                       ->is_segments_overlapping());
 
     _tablet_reader_params.direct_mode = _aggregation || single_version;

Review Comment:
   This line is useless?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [doris] morningman commented on a diff in pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model

Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #11952:
URL: https://github.com/apache/doris/pull/11952#discussion_r952042390


##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -525,6 +529,18 @@ public class SessionVariable implements Serializable, Writable {
     @VariableMgr.VarAttr(name = ENABLE_FUNCTION_PUSHDOWN)
     public boolean enableFunctionPushdown;
 
+    /**
+     * For debugg purpose, dont' merge unique key and agg key when reading data.
+     */
+    @VariableMgr.VarAttr(name = SKIP_STORAGE_ENGINE_MERGE)
+    private boolean skipStorageEngineMerge = false;

Review Comment:
   If this is true, can we select __DELETE_SIGN__ directly?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [doris] jacktengg commented on a diff in pull request #11952: [improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model

Posted by GitBox <gi...@apache.org>.
jacktengg commented on code in PR #11952:
URL: https://github.com/apache/doris/pull/11952#discussion_r966710752


##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -525,6 +529,18 @@ public class SessionVariable implements Serializable, Writable {
     @VariableMgr.VarAttr(name = ENABLE_FUNCTION_PUSHDOWN)
     public boolean enableFunctionPushdown;
 
+    /**
+     * For debugg purpose, dont' merge unique key and agg key when reading data.
+     */
+    @VariableMgr.VarAttr(name = SKIP_STORAGE_ENGINE_MERGE)
+    private boolean skipStorageEngineMerge = false;
+
+    /**
+     * For debugg purpose, skip delte predicate when reading data.
+     */
+    @VariableMgr.VarAttr(name = SKIP_DELETE_PREDICATE)
+    private boolean skipDeletePredicate = false;

Review Comment:
   updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org