You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/06/17 13:50:11 UTC
[GitHub] [doris] TurboWay opened a new pull request, #10236: [feature] add python-doris
TurboWay opened a new pull request, #10236:
URL: https://github.com/apache/doris/pull/10236
# Proposed changes
python for apache-doris
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (No)
2. Has unit tests been added: (Yes)
3. Has document been added or modified: (Yes)
4. Does it need to update dependencies: (No)
5. Are there any changes that cannot be rolled back: (No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
adonis0147 commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1161157332
> > Hi @TurboWay , `mysqlclient` was already installed. Maybe you should add the dependency to `setup.py`.
>
> `setup.py` already had this dependency, but , it's confused that it doesn't work in your case.
Hi @TurboWay , you can build the docker image by the following `Dockerfile`.
```Dockerfile
FROM ubuntu:latest
RUN apt update && apt upgrade --yes
RUN apt install --yes python-is-python3 python3-pip libmysqlclient-dev
ENTRYPOINT ["/bin/bash"]
```
```shell
# Build the image
docker build -t doris_client .
# Run the container
docker run -it doris_client
# In the container, install the DorisClient
pip install DorisClient
# Test the package by python shell
import DorisClient
```
You will reproduce the issue.
A possible solution is to add `pymysql` to `setup.py`, but I didn't test it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] TurboWay commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
TurboWay commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1166789709
HI @adonis0147 , the PR guide to [DorisClient](https://github.com/TurboWay/DorisClient) is commited.
Pip is a common and simple usage. But sometimes, we may not want to rely too much on external packages, and will use the source code directly, so I think it is necessary that add the source code to the repository.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] TurboWay commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
TurboWay commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1159029300
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] TurboWay commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
TurboWay commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1166301613
HI @Gabriel39 ，Thanks for your suggestion.
```DorisClient``` is a small wheel, which is mainly used to make it easier to use streamload and execute Doris SQL in Python. This is very practical in most scenarios.
As for data analysis, in fact, we usually use the famous wheel [```pandas```](https://github.com/pandas-dev/pandas) in Python, which is excellent and also applicable to Doris, because Doris supports the MySQL protocol.
At last, if it is a very large amount of data, it may be a better way to directly use SQL to analyze it in Doris.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Gabriel39 commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
Gabriel39 commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1158919263
Thanks @TurboWay, this feature is very practical. Before reviewing this PR, I noticed that you didn't include a **necessary**
license header for each file. Besides, I think you'd better to translate all comments and documents to English.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] TurboWay commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
TurboWay commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1160233690
> If someone else in community is interested in this tool and he (or she) modifies it, are there some ways to publish this tool in [PyPI](https://pypi.org/project/DorisClient/) ? Maybe some guides to install it manually should be added in documents.
if anyone have some idea to optimize this tool, please make a PR to (DorisClient)[https://github.com/TurboWay/DorisClient], i will glad to merge and publish in PyPI[https://pypi.org/project/DorisClient/]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] TurboWay commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
TurboWay commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1160222457
> Hi @TurboWay , it seems that some dependencies are not installed automatically after I ran `pip install DorisClient`.
>
> 
It seems that mysqlclient was not successfully installed，you can run ```pip install mysqlclient``` to make it work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
adonis0147 commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1166794554
> HI @adonis0147 , the PR guide to [DorisClient](https://github.com/TurboWay/DorisClient) is commited.
>
> Pip is a common and simple usage. But sometimes, we may not want to rely too much on external packages, and will use the source code directly, so I think it is necessary that add the source code to the repository.
In the future, how to maintain this copied source code? I don't think everyone will notice to contribute to the original repository. They may just modify these code and submit a pr to Doris repository only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
adonis0147 commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1166314957
I have some opinions about this pr:
1. It is better to add some documents to official website to guide users to use this Python API to do some jobs like stream load rather than add the source code to the repository. If someone else in community modifies the code, he (or she) may not notice to contribute the code to the upstream [DorisClient](https://github.com/TurboWay/DorisClient). It is hard to maintain duplicate code. You (@TurboWay ) can left the information in documents to guide users to give feedback to [DorisClient](https://github.com/TurboWay/DorisClient).
2. About performance, using [mysqlclient](https://pypi.org/project/mysqlclient/) instead of [PyMySQL](https://pypi.org/project/PyMySQL/) may speed the code up due to the C extension implementation.
This pr inspires me that we can implement a python interactive tool to manage our cluster or do some jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
adonis0147 commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1160090968
If someone else in community is interested in this tool and he (or she) modifies it, are there some ways to publish this tool in [PyPI](https://pypi.org/project/DorisClient/) ? Maybe some guides to install it manually should be added in documents.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
adonis0147 commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1160083049
Hi @TurboWay , it seems that some dependencies are not installed automatically after I ran `pip install DorisClient`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Gabriel39 commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
Gabriel39 commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1166217354
Hi @TurboWay , I think this will be a huge feature for Doris. I'm interested in this and I have some thoughts:
1. I think python client will benefit most data engineers. However, if we decide to support more for them, DataFrame is an essential API we have to support but it has not been done in this PR.
2. For data analysis, a python client is not enough because it has poor effecience for large amount data. So we still need a effecient communication protocol (maybe RPC) between Doris BE server and client.
I think these two are essential and important. Are you interested for a further diving in this way? It's a really wonderful feature and welcome you to contribute more for this and I could offer some helps for you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
adonis0147 commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1160237646
Hi @TurboWay , `mysqlclient` was already installed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] TurboWay commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
TurboWay commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1166951444
> > HI @adonis0147 , the PR guide to [DorisClient](https://github.com/TurboWay/DorisClient) is commited.
> > Pip is a common and simple usage. But sometimes, we may not want to rely too much on external packages, and will use the source code directly, so I think it is necessary that add the source code to the repository.
>
> In the future, how to maintain this copied source code? I don't think everyone will notice to contribute to the original repository. They may just modify these code and submit a pr to Doris repository only.
Maybe I'll check it once in a while, if that happens. However, I think this small wheel should be stable and lightweight, and there should not be much change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] TurboWay commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
TurboWay commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1161322169
>
ok，already fixed it。 ```pymysql``` depend on ``` PyMySQL``` but not ``` mysqlclient ```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] TurboWay commented on pull request #10236: [feature] add python-doris
Posted by GitBox <gi...@apache.org>.
TurboWay commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1160255020
> Hi @TurboWay , `mysqlclient` was already installed. Maybe you should add the dependency to `setup.py`.
setup.py already has this dependency, but , its confuse that it doesn't work in your case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10236: [feature] add python-doris
Posted by github-actions.
github-actions[bot] commented on PR #10236:
URL: https://github.com/apache/doris/pull/10236#issuecomment-1411247473
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] closed pull request #10236: [feature] add python-doris
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #10236: [feature] add python-doris
URL: https://github.com/apache/doris/pull/10236
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org