You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by "zhugezifang (via GitHub)" <gi...@apache.org> on 2023/05/31 12:04:53 UTC
[GitHub] [zeppelin] zhugezifang opened a new pull request, #4611: [ZEPPELIN-5915]improve query efficiency:deal complex sql
zhugezifang opened a new pull request, #4611:
URL: https://github.com/apache/zeppelin/pull/4611
### What is this PR for?
improve query efficiency
Design Document: https://docs.google.com/document/d/1wruK0ZZ0XiriYOraFa5WYSz531pcsCpJeIBmne57fJY/edit?usp=sharing
### What type of PR is it?
Improvement
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-5915
### How should this be tested?
* CI
### Questions:
* Does the license files need to update? yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5915]improve query efficiency [zeppelin]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #4611: [ZEPPELIN-5915]improve query efficiency
URL: https://github.com/apache/zeppelin/pull/4611
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhugezifang commented on pull request #4611: [ZEPPELIN-5915]improve query efficiency
Posted by "zhugezifang (via GitHub)" <gi...@apache.org>.
zhugezifang commented on PR #4611:
URL: https://github.com/apache/zeppelin/pull/4611#issuecomment-1573214026
hi @Reamer @jongyoul @huage1994 ,could you help me to review this pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5915]improve query efficiency [zeppelin]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4611:
URL: https://github.com/apache/zeppelin/pull/4611#issuecomment-3071707138
This pull request has been inactive for over a year. If no further activity occurs within the next 30 days, it will be automatically closed. If you believe this PR is still relevant, please feel free to leave a comment or make an update. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] jongyoul commented on pull request #4611: [ZEPPELIN-5915]improve query efficiency
Posted by "jongyoul (via GitHub)" <gi...@apache.org>.
jongyoul commented on PR #4611:
URL: https://github.com/apache/zeppelin/pull/4611#issuecomment-1582577943
> and the data cache is really for hive like this large data database (not for mysql), as we know , hive sql will tranlate to mapreduce job， complex hive sql have shuffle and reducer ,it cost a lot of time
I agree that Zeppelin could help to reduce resource usage but I, as one of the maintainers, believe that we shouldn't change any user behaviors and mechanisms working as it is. Your approach might be better for some cases but it could have a kind of hidden steps and it could have another problem. If you would like to do it really, I believe that you can make your/your company's own interpreter. It can be merged into this repository but we, as reviewers, should consider not to make significant side effects with the highest priority.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on pull request #4611: [ZEPPELIN-5915]improve query efficiency
Posted by "Reamer (via GitHub)" <gi...@apache.org>.
Reamer commented on PR #4611:
URL: https://github.com/apache/zeppelin/pull/4611#issuecomment-1582000098
A few thoughts about your work:
- All functionality is located in the JDBC interpreter. All new dependencies and classes should also be located there.
- It seems that in case of a complex query you modify the dataset with the JDBC interpreter at the source. With your current implementation, the user is not aware of this right away.
- You want to cache complex queries with an LRUCache. Depending on the size of the cache, queries may not be removed from the cache and the user may see stale data. I think a time based cache would be more appropriate here.
- Have you ever heard of PreparedStatements?
- At first view your configuration options are only JDBC interpreter configuration, therefore the configuration should be stored only in the JDBC interpreter and not as global ZeppelinConfiguration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhugezifang commented on pull request #4611: [ZEPPELIN-5915]improve query efficiency
Posted by "zhugezifang (via GitHub)" <gi...@apache.org>.
zhugezifang commented on PR #4611:
URL: https://github.com/apache/zeppelin/pull/4611#issuecomment-1582067198
> A few thoughts about your work:
>
> * All functionality is located in the JDBC interpreter. All new dependencies and classes should also be located there.
> * It seems that in case of a complex query you modify the dataset with the JDBC interpreter at the source. With your current implementation, the user is not aware of this right away.
> * You want to cache complex queries with an LRUCache. Depending on the size of the cache, queries may not be removed from the cache and the user may see stale data. I think a time based cache would be more appropriate here.
> * Have you ever heard of PreparedStatements?
> * At first view your configuration options are only JDBC interpreter configuration, therefore the configuration should be stored only in the JDBC interpreter and not as global ZeppelinConfiguration.
and could you help me to continue complete the feature of sql debug https://github.com/apache/zeppelin/pull/4598
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhugezifang commented on pull request #4611: [ZEPPELIN-5915]improve query efficiency
Posted by "zhugezifang (via GitHub)" <gi...@apache.org>.
zhugezifang commented on PR #4611:
URL: https://github.com/apache/zeppelin/pull/4611#issuecomment-1582091105
> A few thoughts about your work:
>
> * All functionality is located in the JDBC interpreter. All new dependencies and classes should also be located there.
> * It seems that in case of a complex query you modify the dataset with the JDBC interpreter at the source. With your current implementation, the user is not aware of this right away.
> * You want to cache complex queries with an LRUCache. Depending on the size of the cache, queries may not be removed from the cache and the user may see stale data. I think a time based cache would be more appropriate here.
> * Have you ever heard of PreparedStatements?
> * At first view your configuration options are only JDBC interpreter configuration, therefore the configuration should be stored only in the JDBC interpreter and not as global ZeppelinConfiguration.
and the data cache is really for hive like this large data database (not for mysql), as we know ,complex hive sql have shuffle and reducer ,it cost a lot of time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhugezifang commented on pull request #4611: [ZEPPELIN-5915]improve query efficiency
Posted by "zhugezifang (via GitHub)" <gi...@apache.org>.
zhugezifang commented on PR #4611:
URL: https://github.com/apache/zeppelin/pull/4611#issuecomment-1582065412
@Reamer hi，firstly thanks for your advice
"All functionality is located in the JDBC interpreter. All new dependencies and classes should also be located there"---- it contains antlr4 ,and it also can be used in the feature in sql debug https://github.com/apache/zeppelin/pull/4598
but i am very sorry , the feature cost you a lot of time ,but it can not be merged
if this feature of sql debug can be merge, the use of antlr can also be used in this feature.
'queries may not be removed from the cache and the user may see stale data' --- this is really a good question，it is i want to do in next step，i create temp table as the data, but if the origin data change or temp table very much ,it need to fix ,so i write the Design Document, the last step is to clean the temp table
https://docs.google.com/document/d/1wruK0ZZ0XiriYOraFa5WYSz531pcsCpJeIBmne57fJY/edit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] jongyoul commented on pull request #4611: [ZEPPELIN-5915]improve query efficiency
Posted by "jongyoul (via GitHub)" <gi...@apache.org>.
jongyoul commented on PR #4611:
URL: https://github.com/apache/zeppelin/pull/4611#issuecomment-1582561003
Sorry for the late and short comment but how about making debug paragraph as a new interpreter? I mean, for instance, `%jdbc.debug`. Debugging jdbc paragraph itself is reasonable but I'm worried that this feature would be included in ZeppelinServer because this is only for jdbc interpreter and it won't work for other interpereters. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhugezifang commented on pull request #4611: [ZEPPELIN-5915]improve query efficiency
Posted by "zhugezifang (via GitHub)" <gi...@apache.org>.
zhugezifang commented on PR #4611:
URL: https://github.com/apache/zeppelin/pull/4611#issuecomment-1584017096
> > and the data cache is really for hive like this large data database (not for mysql), as we know , hive sql will tranlate to mapreduce job， complex hive sql have shuffle and reducer ,it cost a lot of time
>
> I agree that Zeppelin could help to reduce resource usage but I, as one of the maintainers, believe that we shouldn't change any user behaviors and mechanisms working as it is. Your approach might be better for some cases but it could have a kind of hidden steps and it could have another problem. If you would like to do it really, I believe that you can make your/your company's own interpreter. It can be merged into this repository but we, as reviewers, should consider not to make significant side effects with the highest priority.
ok, thanks for your advice, and could you help to review this pr of sql debug, https://github.com/apache/zeppelin/pull/4598
is it suitable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5915]improve query efficiency [zeppelin]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4611:
URL: https://github.com/apache/zeppelin/pull/4611#issuecomment-3194074905
This pull request has been automatically closed due to prolonged inactivity (over one year without updates). If you feel this was done in error or would like to continue the discussion, feel free to reopen it. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org