You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "trungda (via GitHub)" <gi...@apache.org> on 2024/01/19 22:32:30 UTC
[PR] use tokenizer [arrow-datafusion]
trungda opened a new pull request, #8918:
URL: https://github.com/apache/arrow-datafusion/pull/8918
## Which issue does this PR close?
https://github.com/apache/arrow-datafusion/issues/8701
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
## Rationale for this change
More user-friendly and readability when using `datafusion-cli`.
## What changes are included in this PR?
Add SQL syntax highlighting capability for `datafusion-cli`: `SyntaxHighlighter`.
## Are these changes tested?
1. Added unit tests to validate the ANSI string generated;
2. Manually tested.

## Are there any user-facing changes?
Users of datafusion-cli.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add syntax highlight to datafusion-cli [arrow-datafusion]
Posted by "trungda (via GitHub)" <gi...@apache.org>.
trungda commented on PR #8918:
URL: https://github.com/apache/arrow-datafusion/pull/8918#issuecomment-1903363187
Thanks @alamb ! I've updated the license + changed the color to be lighter as suggested.
> That would be great -- thank you @trungda (this is somewhat related); https://github.com/sqlparser-rs/sqlparser-rs/pull/1094
This is great! I'll look into that. It'd be easier if somehow we know the position in the original string where the tokernization fails just so we can just copy from that position to the highlighted string.
> Another feature that would be neat would be a CLI flag to enable/disable the highlighting so people who find the color scheme not to their liking could turn it off
Sounds great! Let me look into that after this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add syntax highlight to datafusion-cli [arrow-datafusion]
Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on PR #8918:
URL: https://github.com/apache/arrow-datafusion/pull/8918#issuecomment-1902486934
Awesome feature! Thanks, @trungda 👍 . I am wondering if could make the default color lighter because the current version is hard to track. Maybe you could consider the DuckDB setting.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add syntax highlight to datafusion-cli [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #8918:
URL: https://github.com/apache/arrow-datafusion/pull/8918#issuecomment-1902269097
<img width="87" alt="image" src="https://github.com/apache/arrow-datafusion/assets/10247224/10c56c78-4277-4b71-8612-6564638bea38">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add syntax highlight to datafusion-cli [arrow-datafusion]
Posted by "trungda (via GitHub)" <gi...@apache.org>.
trungda commented on PR #8918:
URL: https://github.com/apache/arrow-datafusion/pull/8918#issuecomment-1901243443
The current implementation is not the best. If the input string is invalid, e.g., unterminated: `select * from tab_1 where col_1 = '`, the whole string will be _un_highlighted. I think a better approach is to make change to `sqlparser` crate to expose a token iterator just so we can keep highlighting as we scan through the original string, and only stop when we encounter a failure. Something along this line: https://github.com/sqlparser-rs/sqlparser-rs/blob/main/src/tokenizer.rs#L555-L558.
I am happy to go down that path too. Let me know :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add syntax highlight to datafusion-cli [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #8918:
URL: https://github.com/apache/arrow-datafusion/pull/8918#issuecomment-1902268875
<img width="130" alt="image" src="https://github.com/apache/arrow-datafusion/assets/10247224/751ee2f8-d8ca-4788-9567-c38d4e0aac17">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add syntax highlight to datafusion-cli [arrow-datafusion]
Posted by "trungda (via GitHub)" <gi...@apache.org>.
trungda commented on PR #8918:
URL: https://github.com/apache/arrow-datafusion/pull/8918#issuecomment-1902271454
> <img alt="image" width="87" src="https://private-user-images.githubusercontent.com/10247224/298308538-10c56c78-4277-4b71-8612-6564638bea38.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDU3ODY3MDIsIm5iZiI6MTcwNTc4NjQwMiwicGF0aCI6Ii8xMDI0NzIyNC8yOTgzMDg1MzgtMTBjNTZjNzgtNDI3Ny00YjcxLTg2MTItNjU2NDYzOGJlYTM4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAxMjAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMTIwVDIxMzMyMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTM1ZWZmZDA0MzlkMjBmNWYzMDNhZThmMWI3YWQwODExZDIzMGI1NDUzYmQ0YWQ0MjlkYzhhOTFiNjJjNGU3YjQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ZQ42izYYG11pZ4KSJSX8R2jwuXeTx4ElDrn0M14uTP4">
Thanks for the feedback @comphead . What is your terminal setup? I am not able to repro yet. The current approach is to take the whole input string and tokenize the input as a whole, the arrow keys shouldn't matter.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add syntax highlight to datafusion-cli [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8918:
URL: https://github.com/apache/arrow-datafusion/pull/8918#issuecomment-1902682863
> I am happy to go down that path too. Let me know :)
That would be great -- thank you @trungda (this is somewhat related); https://github.com/sqlparser-rs/sqlparser-rs/pull/1094
> On that note, can we add datafusion-cli to members attribute in [workspace] just so it will be recognized by the rust-analyzer plugin in IDEs?
The reason it is not part of the workspace is that as `datafusion-cli` we want to release a binary and thus have the versions locked (via a checked in `Cargo.lock`) However, for the `datafusion` crates we release them as a library where having Cargo.lock is not the recommended.
This is not clear I will add it to the README
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add syntax highlight to datafusion-cli [arrow-datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #8918:
URL: https://github.com/apache/arrow-datafusion/pull/8918#discussion_r1461137404
##########
datafusion-cli/src/highlighter.rs:
##########
@@ -0,0 +1,126 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apndache.org/licenses/LICENSE-2.0
Review Comment:
```suggestion
// http://www.apache.org/licenses/LICENSE-2.0
```
typo in license
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add syntax highlight to datafusion-cli [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #8918:
URL: https://github.com/apache/arrow-datafusion/pull/8918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add syntax highlight to datafusion-cli [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8918:
URL: https://github.com/apache/arrow-datafusion/pull/8918#discussion_r1460977665
##########
datafusion-cli/src/highlighter.rs:
##########
@@ -0,0 +1,126 @@
+// Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
For some reason the RAT check fails for this file: https://github.com/apache/arrow-datafusion/actions/runs/7592786554/job/20702838429?pr=8918
I think that can typically be fixed by just copy/pasting the license text / comment from another file
##########
datafusion-cli/src/highlighter.rs:
##########
@@ -0,0 +1,126 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apndache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The syntax highlighter.
+
+use std::{
+ borrow::Cow::{self, Borrowed},
+ fmt::Display,
+};
+
+use datafusion::sql::sqlparser::{
+ dialect::{dialect_from_str, Dialect, GenericDialect},
+ keywords::Keyword,
+ tokenizer::{Token, Tokenizer},
+};
+use rustyline::highlight::Highlighter;
+
+/// The syntax highlighter.
+pub struct SyntaxHighlighter {
+ dialect: Box<dyn Dialect>,
+}
+
+impl SyntaxHighlighter {
+ pub fn new(dialect: &str) -> Self {
+ let dialect = match dialect_from_str(dialect) {
+ Some(dialect) => dialect,
+ None => Box::new(GenericDialect {}),
+ };
+ Self { dialect }
+ }
+}
+
+impl Highlighter for SyntaxHighlighter {
+ fn highlight<'l>(&self, line: &'l str, _: usize) -> Cow<'l, str> {
+ let mut out_line = String::new();
+
+ // `with_unescape(false)` since we want to rebuild the original string.
Review Comment:
💯
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add syntax highlight to datafusion-cli [arrow-datafusion]
Posted by "trungda (via GitHub)" <gi...@apache.org>.
trungda commented on code in PR #8918:
URL: https://github.com/apache/arrow-datafusion/pull/8918#discussion_r1461412266
##########
datafusion-cli/src/highlighter.rs:
##########
@@ -0,0 +1,126 @@
+// Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
Thanks! There was a typo in the license as pointed about by @Jefffrey . Let's see if it's green now :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org