You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "danigiri (via GitHub)" <gi...@apache.org> on 2024/10/15 10:36:17 UTC

[PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]

danigiri opened a new pull request, #33258:
URL: https://github.com/apache/shardingsphere/pull/33258

   As part of hacktoberfest, this is a POC for adding more parsing logic to Doris support. If this is successful, I will look at the other Doris SQL keywords in subsequent PRs. Starts to address #31508.
   
   I understand that I cannot add the labels to the PR and maintainers have to do it.
   
   ### Changes proposed in this pull request:
    - Made g4 changes, with a structure in preparation for adding more functions like unary bitwise
    - Includes test case additions to XML
    - Not sure if the change to `getAllDatabaseTypes` is really needed, can roll it back
   
   ### Next:
   - Just realised I modified the `BIT_XOR` aggregation keyword instead of creating a new one. I can fix this quickly in a new commit in this PR
   
   ### Comments:
   - Doris requires AVX2 hardware which I do not have local access to (only AVX), it lets you compile without AVX2 support, but it's been 24h compilation time and not finished yet (or in an infinite loop).
   - I launched a DigitalOcean droplet ($) instead which let me test the SQL.
   - I am not sure if other backend changes are required once parsing support is added
   - Not sure which doc changes are needed or where
   
   ---
   
   Before committing this PR, I'm sure that I have checked the following options:
   - [ ✅] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project. _As much as I understood it_
   - [ ✅] I have self-reviewed the commit code.
   - [ ✅] I have (or in comment I request) added corresponding labels for the pull request.
   - [✅] I have passed maven check locally : `./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e`.
   - [] I have made corresponding changes to the documentation.
   - [✅] I have added corresponding unit tests for my changes.
   - [✅] `mvn spotless:apply -Pcheck`
   - [✅] `mvn test -Dtest=InternalDorisParserIT` in `shardingsphere/parser/sql/dialect/doris`
   - [✅] Executed test case in Doris v2.0.3 `SELECT BITXOR(3,5)`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]

Posted by "danigiri (via GitHub)" <gi...@apache.org>.
danigiri commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2419202100

   Cool, thx. I am going to give it a go in my upcoming PR, thanks for the clrifications.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]

Posted by "iamhucong (via GitHub)" <gi...@apache.org>.
iamhucong commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2416201456

   LGTM.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]

Posted by "iamhucong (via GitHub)" <gi...@apache.org>.
iamhucong commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2418320288

   It looks like MySQL only supports BIT_XOR() and is slightly different from Doris. Maybe you can add a mark with the differential codes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]

Posted by "iamhucong (via GitHub)" <gi...@apache.org>.
iamhucong commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2419193460

   Great, thanks for checking that out!
   
   https://shardingsphere.apache.org/community/en/involved/conduct/code/#maintenance-conduct
   > Difference code markup syntax, when adding, replace {DatabaseType} with the uppercase name of the database type, for example: DORIS.
   >
   >New syntax: //{DatabaseType} ADDED BEGIN and // {DatabaseType} ADDED END;
   >Modified syntax: //{DatabaseType} CHANGED BEGIN and // {DatabaseType} CHANGED END.
   
   I mean this comment mark. Try searching for `// DORIS ADDED BEGIN` as a reference.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]

Posted by "danigiri (via GitHub)" <gi...@apache.org>.
danigiri commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2426115006

   Compiling the just released doris 3.0 and will look at the next keywords


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]

Posted by "danigiri (via GitHub)" <gi...@apache.org>.
danigiri commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2417174030

   looks like MySQL implements the same operation with `^` (https://dev.mysql.com/doc/refman/8.4/en/bit-functions.html)
   ```
   SELECT 3 ^ 5;
   +-------+
   | 3 ^ 5    |
   +-------+
   |          6 |
   +-------+
   SELECT BITXOR(3,5);
   ERROR 1305 (42000): FUNCTION mysql.BITXOR does not exist
   ```
   
   which is already present in `/dialect_mysql/src/main/antlr4/imports/mysql/BaseRule.g4`
   ```
       | bitExpr CARET_ bitExpr
   ```
   From what I understand, this means no further action is needed?
   
   I will look at the `INSTR` in the meantime


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]

Posted by "danigiri (via GitHub)" <gi...@apache.org>.
danigiri commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2419111150

   cc @iamhucong (see my reply above, thx)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]

Posted by "danigiri (via GitHub)" <gi...@apache.org>.
danigiri commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2416279523

   
   Thanks for the review and comments, will  do next:
   - update release notes
   - check MySQL BITXOR support
   
   Also
   - I did not originally understand https://shardingsphere.apache.org/community/en/involved/conduct/code/#maintenance-conduct sorry about that
   - from what I understand, I would also need to update MySQL parsing support for BITXOR as well (I can look into this too)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]

Posted by "danigiri (via GitHub)" <gi...@apache.org>.
danigiri commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2419108745

   > It looks like MySQL only supports BIT_XOR() and is slightly different from Doris. Maybe you can add a mark with the differential codes.
   
   Not exactly, they are in fact relatively different, here's a breakdown of the situation:
   
   #### MySQL BIT_XOR()
   This is an aggregation function https://dev.mysql.com/doc/refman/8.4/en/aggregate-functions.html
   ```
   SELECT BIT_XOR(value) FROM (VALUES ROW(3), ROW(5)) AS t(value);                                                                                                                                                                         
   +----------------+
   | BIT_XOR(value) |
   +----------------+
   |              6 |
   +----------------+
   1 row in set (0.01 sec)
   ```
   
   #### ^ in MySQL
   This is a bitwise expression https://dev.mysql.com/doc/refman/8.4/en/bit-functions.html
   ```
   SELECT 3^5;
   +-----+
   | 3^5 |
   +-----+
   |   6 |
   +-----+
   1 row in set (0.00 sec)
   ```
   
   #### Doris BITXOR()
   This is a bitwise function https://doris.apache.org/docs/sql-manual/sql-functions/bitwise-functions/bitxor/ which is equivalent to the MySQL `^` expression
   ```
   SELECT BITXOR(3,5);
   +---------+
   |  3 ^ 5) |
   +---------+
   |       6 |
   +---------+
   1 row in set (0.05 sec)
   ```
   
   #### Doris GROUP_BIT_XOR
   This is an aggregation function https://doris.apache.org/docs/1.2/sql-manual/sql-functions/aggregate-functions/group_bit_xor/ equivalent to the MySQL `bit_xor` aggregation function
   ```
   WITH values_cte AS (     
   SELECT 3 AS value  UNION ALL  SELECT 5 AS value ) 
   SELECT GROUP_BIT_XOR(value) FROM values_cte;
   +----------------------+
   | group_bit_xor(value) |
   +----------------------+
   |                    6 |
   +----------------------+
   1 row in set (0.30 sec)
   ```
   
   
   In summary:
   
   |                    |  MySQL       | Doris                         |
   |------------|-------------|---------------------|
   |  Aggr          | `BIT_XOR`  | `GROUP_BIT_XOR` |
   |  Expr/func  |  `^`              | `BITXOR`                 |
   
   
   Given this, does it change the approach? 
   
   I am not sure what you mean by
   
   > add a mark with the differential codes.
   
   if you have a commit / PR I could use as an example that would be super. Do you mean a comment in .g4 or something else?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]

Posted by "iamhucong (via GitHub)" <gi...@apache.org>.
iamhucong merged PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]

Posted by "strongduanmu (via GitHub)" <gi...@apache.org>.
strongduanmu commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2416217434

   Hi @danigiri, can you update the release note for 5.5.2-SNAPSHOT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]

Posted by "iamhucong (via GitHub)" <gi...@apache.org>.
iamhucong commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2416234140

   https://shardingsphere.apache.org/community/en/involved/conduct/code/#maintenance-conduct
   @danigiri Could you please verify if MySQL supports BITXOR? You can refer to this document for further information.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org