You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "danigiri (via GitHub)" <gi...@apache.org> on 2024/10/15 10:36:17 UTC
[PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]
danigiri opened a new pull request, #33258:
URL: https://github.com/apache/shardingsphere/pull/33258
As part of hacktoberfest, this is a POC for adding more parsing logic to Doris support. If this is successful, I will look at the other Doris SQL keywords in subsequent PRs. Starts to address #31508.
I understand that I cannot add the labels to the PR and maintainers have to do it.
### Changes proposed in this pull request:
- Made g4 changes, with a structure in preparation for adding more functions like unary bitwise
- Includes test case additions to XML
- Not sure if the change to `getAllDatabaseTypes` is really needed, can roll it back
### Next:
- Just realised I modified the `BIT_XOR` aggregation keyword instead of creating a new one. I can fix this quickly in a new commit in this PR
### Comments:
- Doris requires AVX2 hardware which I do not have local access to (only AVX), it lets you compile without AVX2 support, but it's been 24h compilation time and not finished yet (or in an infinite loop).
- I launched a DigitalOcean droplet ($) instead which let me test the SQL.
- I am not sure if other backend changes are required once parsing support is added
- Not sure which doc changes are needed or where
---
Before committing this PR, I'm sure that I have checked the following options:
- [ ✅] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project. _As much as I understood it_
- [ ✅] I have self-reviewed the commit code.
- [ ✅] I have (or in comment I request) added corresponding labels for the pull request.
- [✅] I have passed maven check locally : `./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e`.
- [] I have made corresponding changes to the documentation.
- [✅] I have added corresponding unit tests for my changes.
- [✅] `mvn spotless:apply -Pcheck`
- [✅] `mvn test -Dtest=InternalDorisParserIT` in `shardingsphere/parser/sql/dialect/doris`
- [✅] Executed test case in Doris v2.0.3 `SELECT BITXOR(3,5)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]
Posted by "danigiri (via GitHub)" <gi...@apache.org>.
danigiri commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2419202100
Cool, thx. I am going to give it a go in my upcoming PR, thanks for the clrifications.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]
Posted by "iamhucong (via GitHub)" <gi...@apache.org>.
iamhucong commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2416201456
LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]
Posted by "iamhucong (via GitHub)" <gi...@apache.org>.
iamhucong commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2418320288
It looks like MySQL only supports BIT_XOR() and is slightly different from Doris. Maybe you can add a mark with the differential codes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]
Posted by "iamhucong (via GitHub)" <gi...@apache.org>.
iamhucong commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2419193460
Great, thanks for checking that out!
https://shardingsphere.apache.org/community/en/involved/conduct/code/#maintenance-conduct
> Difference code markup syntax, when adding, replace {DatabaseType} with the uppercase name of the database type, for example: DORIS.
>
>New syntax: //{DatabaseType} ADDED BEGIN and // {DatabaseType} ADDED END;
>Modified syntax: //{DatabaseType} CHANGED BEGIN and // {DatabaseType} CHANGED END.
I mean this comment mark. Try searching for `// DORIS ADDED BEGIN` as a reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]
Posted by "danigiri (via GitHub)" <gi...@apache.org>.
danigiri commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2426115006
Compiling the just released doris 3.0 and will look at the next keywords
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]
Posted by "danigiri (via GitHub)" <gi...@apache.org>.
danigiri commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2417174030
looks like MySQL implements the same operation with `^` (https://dev.mysql.com/doc/refman/8.4/en/bit-functions.html)
```
SELECT 3 ^ 5;
+-------+
| 3 ^ 5 |
+-------+
| 6 |
+-------+
SELECT BITXOR(3,5);
ERROR 1305 (42000): FUNCTION mysql.BITXOR does not exist
```
which is already present in `/dialect_mysql/src/main/antlr4/imports/mysql/BaseRule.g4`
```
| bitExpr CARET_ bitExpr
```
From what I understand, this means no further action is needed?
I will look at the `INSTR` in the meantime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]
Posted by "danigiri (via GitHub)" <gi...@apache.org>.
danigiri commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2419111150
cc @iamhucong (see my reply above, thx)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]
Posted by "danigiri (via GitHub)" <gi...@apache.org>.
danigiri commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2416279523
Thanks for the review and comments, will do next:
- update release notes
- check MySQL BITXOR support
Also
- I did not originally understand https://shardingsphere.apache.org/community/en/involved/conduct/code/#maintenance-conduct sorry about that
- from what I understand, I would also need to update MySQL parsing support for BITXOR as well (I can look into this too)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]
Posted by "danigiri (via GitHub)" <gi...@apache.org>.
danigiri commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2419108745
> It looks like MySQL only supports BIT_XOR() and is slightly different from Doris. Maybe you can add a mark with the differential codes.
Not exactly, they are in fact relatively different, here's a breakdown of the situation:
#### MySQL BIT_XOR()
This is an aggregation function https://dev.mysql.com/doc/refman/8.4/en/aggregate-functions.html
```
SELECT BIT_XOR(value) FROM (VALUES ROW(3), ROW(5)) AS t(value);
+----------------+
| BIT_XOR(value) |
+----------------+
| 6 |
+----------------+
1 row in set (0.01 sec)
```
#### ^ in MySQL
This is a bitwise expression https://dev.mysql.com/doc/refman/8.4/en/bit-functions.html
```
SELECT 3^5;
+-----+
| 3^5 |
+-----+
| 6 |
+-----+
1 row in set (0.00 sec)
```
#### Doris BITXOR()
This is a bitwise function https://doris.apache.org/docs/sql-manual/sql-functions/bitwise-functions/bitxor/ which is equivalent to the MySQL `^` expression
```
SELECT BITXOR(3,5);
+---------+
| 3 ^ 5) |
+---------+
| 6 |
+---------+
1 row in set (0.05 sec)
```
#### Doris GROUP_BIT_XOR
This is an aggregation function https://doris.apache.org/docs/1.2/sql-manual/sql-functions/aggregate-functions/group_bit_xor/ equivalent to the MySQL `bit_xor` aggregation function
```
WITH values_cte AS (
SELECT 3 AS value UNION ALL SELECT 5 AS value )
SELECT GROUP_BIT_XOR(value) FROM values_cte;
+----------------------+
| group_bit_xor(value) |
+----------------------+
| 6 |
+----------------------+
1 row in set (0.30 sec)
```
In summary:
| | MySQL | Doris |
|------------|-------------|---------------------|
| Aggr | `BIT_XOR` | `GROUP_BIT_XOR` |
| Expr/func | `^` | `BITXOR` |
Given this, does it change the approach?
I am not sure what you mean by
> add a mark with the differential codes.
if you have a commit / PR I could use as an example that would be super. Do you mean a comment in .g4 or something else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]
Posted by "iamhucong (via GitHub)" <gi...@apache.org>.
iamhucong merged PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]
Posted by "strongduanmu (via GitHub)" <gi...@apache.org>.
strongduanmu commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2416217434
Hi @danigiri, can you update the release note for 5.5.2-SNAPSHOT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added BITXOR to Doris parsing support (#31508) [shardingsphere]
Posted by "iamhucong (via GitHub)" <gi...@apache.org>.
iamhucong commented on PR #33258:
URL: https://github.com/apache/shardingsphere/pull/33258#issuecomment-2416234140
https://shardingsphere.apache.org/community/en/involved/conduct/code/#maintenance-conduct
@danigiri Could you please verify if MySQL supports BITXOR? You can refer to this document for further information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org