You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/04/24 14:33:36 UTC
[GitHub] [dolphinscheduler] jieguangzhou opened a new issue, #9725: [Discussion][Machine Learning] Support AI task and the open source project about MLops
jieguangzhou opened a new issue, #9725:
URL: https://github.com/apache/dolphinscheduler/issues/9725
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
I have seen a Machine Learning Platform post on Medium. The post talk about Lizhi Machine Learning Platform&Apache DolphinScheduler.
https://medium.com/@DolphinScheduler/a-formidable-combination-of-lizhi-machine-learning-platform-dolphinscheduler-creates-new-paradigm-e445938f1af
Like this, I have a try to do something like this.
Figure 1 shows the training workflow startup screen
<img width="496" alt="image" src="https://user-images.githubusercontent.com/31528124/164979694-b55493e8-c8e7-4fda-9489-5d23b929cf05.png">
In this workflow, I implemented four algorithms (SVM, LR, LGBM, XGboost) using the API of Sklearn, Lightgbm, and Xgboost.
Every algorithm's parameters can fill in the value of key "params". In this case, the parameters of LGBM is "n_estimators=200;num_leaves=20".
The experiment tracking module is supported by MLFlow.The picture below shows the report of the experiment.
I register the model every time I run it.
When the model is trained, run the deployment workflow. Like this:
<img width="499" alt="image" src="https://user-images.githubusercontent.com/31528124/164981446-4ed0ec0c-6c7b-4ffa-9db4-c839560740fa.png">
We can deploy the version 2 model to the k8s cluster.
And then we can see the deployment and pods
At the same time, we can access the service through the interface.
BTW, we can also connect the training workflow with the deployment workflow as a sub-workflow, like this.
<img width="613" alt="image" src="https://user-images.githubusercontent.com/31528124/164980627-63517bcd-aca0-4e74-b16e-0113381ef279.png">
### What you expected to happen
None
### How to reproduce
None
### Anything else
The above workflow is based on the Shell task. But it is too complex to ml engineer. I hope to write new types of tasks that make them easier for users to use.
The training workflow contains one task. The code is as follows
```
data_path=${data_path}
export MLFLOW_TRACKING_URI=${MLFLOW_TRACKING_URI}
echo $data_path
repo=https://github.com/jieguangzhou/mlflow_sklearn_gallery.git
mlflow run $repo -P algorithm=${algorithm} -P data_path=$data_path -P params="${params}" -P param_file=${param_file} -P model_name=${model_name} --experiment-name=${experiment_name}
echo "training finish"
```
The deployment workflow contains two task.
<img width="639" alt="image" src="https://user-images.githubusercontent.com/31528124/164980832-fd6a524e-daeb-4e93-a62d-3fcbcfe3a74b.png">
The code of the "build docker" workflow is as follows
```
eval $(minikub -p minikube docker-env)
export MLFLOW_TRACKING_URI=${MLFLOW_TRACKING_URI}
image_name=mlflow/${model_name}:${version}
echo $image_name
mlflow models build-docker -m "models:/${model_name}/${version}" -n $image_name --enable-mlserver
```
The code of the "create deployment" workflow which deploys the model to the k8s cluster is as follows
```
version_lower=$(echo "${version}" | tr '[:upper:]' '[:lower:]')
kubectl apply -f - << END
apiVersion: apps/v1
kind: Deployment
metadata:
name: mlflow-${model_name}-$version_lower
spec:
selector:
matchLabels:
app: mlflow
replicas: 3 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: mlflow
spec:
containers:
- name: mlflow-iris
image: mlflow/${model_name}:${version}
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: mlflow-${model_name}-$version_lower
spec:
ports:
- port: 8080
targetPort: 8080
selector:
app: mlflow
END
sleep 5s
kubectl port-forward deployment/mlflow-${model_name}-$version_lower ${deployment_port}:8080
```
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng closed issue #9725: [Discussion][Machine Learning] Support AI task and the open source project about MLops
Posted by GitBox <gi...@apache.org>.
caishunfeng closed issue #9725: [Discussion][Machine Learning] Support AI task and the open source project about MLops
URL: https://github.com/apache/dolphinscheduler/issues/9725
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jieguangzhou commented on issue #9725: [Discussion][Machine Learning] Support AI task and the open source project about MLops
Posted by GitBox <gi...@apache.org>.
jieguangzhou commented on issue #9725:
URL: https://github.com/apache/dolphinscheduler/issues/9725#issuecomment-1119234197
> Hi, we are planning to initiate our exploration of AIOps at [Apache SkyWalking community](https://github.com/apache/skywalking/discussions). Very interesting to see the discussions here.
>
> Now I'm also looking at Dolphinscheduler to handle our workflow orchestration, and for now, we may as well go in the same direction as yours. I feel like integration with MLFlow package functionalities will be a good point to boost ML-developer experience to the next level.
Hi, good to see you join the discussion. I just read about your discussion (https://github.com/apache/skywalking/discussions/8883). It should be a great project.
I think DolphinScheduler will be able to schedule the AIops scenario in the near future. I am enriching its scheduling features in the field of artificial intelligence, and the MVP product is being implemented.
We can keep talking about that. BTW, I might do some experiments with this data set, but I can't access it right now. https://github.com/CloudWise-OpenSource/GAIA-DataSet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jieguangzhou commented on issue #9725: [Discussion][Machine Learning] Support AI task and the open source project about MLops
Posted by GitBox <gi...@apache.org>.
jieguangzhou commented on issue #9725:
URL: https://github.com/apache/dolphinscheduler/issues/9725#issuecomment-1108445809
I add the GridSearch feature to the example like this. That we can search for the best parameters of a model.
<img width="491" alt="image" src="https://user-images.githubusercontent.com/31528124/165078096-54543d05-36db-425f-b988-745af105db03.png">
After the model was trained, we also can see the parameter search report on the MLflow dashboard like this
<img width="2317" alt="image" src="https://user-images.githubusercontent.com/31528124/165079578-d23a2098-8df3-44e0-9863-fb73499f79c0.png">
The value of key "search_params" above is "max_depth=[5, 10];n_estimators=[100, 200]" for lightgbm.
We also can search more params, all of the params we can see In API of the algorithm, for example:
svm: "kernel=['linear', 'poly', 'rbf'];C=[0.5, 1.0]" ;
lr: "penalty=['l1', 'l2'];C=[0.5, 1.0]" ;
xgboost: max_depth=[5, 10];n_estimators=[100, 200]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on issue #9725: [Discussion][Machine Learning] Support AI task and the open source project about MLops
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on issue #9725:
URL: https://github.com/apache/dolphinscheduler/issues/9725#issuecomment-1119243314
Looking great! I am very optimistic about the prospects of this. And as I said in the mail thread, I think machine learning is also another kind of orchestration, and most of the machine learning source data or training samples are from data warehouses or data lakes, which we already supported in the current version. If we DolphinScheduler could support machine learning tasks then users could finish their jobs in one single tool instead of separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jieguangzhou commented on issue #9725: [Discussion][Machine Learning] Support AI task and the open source project about MLops
Posted by GitBox <gi...@apache.org>.
jieguangzhou commented on issue #9725:
URL: https://github.com/apache/dolphinscheduler/issues/9725#issuecomment-1108117248
Is somebody interested in the AI task about DS?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Superskyyy commented on issue #9725: [Discussion][Machine Learning] Support AI task and the open source project about MLops
Posted by GitBox <gi...@apache.org>.
Superskyyy commented on issue #9725:
URL: https://github.com/apache/dolphinscheduler/issues/9725#issuecomment-1119278387
I'll be happy to follow this and provide help. Also happy to integrate and test the outcomes in the new SkyWalking ecosystem AIOps project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #9725: [Discussion][Machine Learning] Support AI task and the open source project about MLops
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #9725:
URL: https://github.com/apache/dolphinscheduler/issues/9725#issuecomment-1107853856
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jieguangzhou commented on issue #9725: [Discussion][Machine Learning] Support AI task and the open source project about MLops
Posted by GitBox <gi...@apache.org>.
jieguangzhou commented on issue #9725:
URL: https://github.com/apache/dolphinscheduler/issues/9725#issuecomment-1108454395
The above work is trying to build an MLops system using dolphinscheduler as an orchestration system.
I think that will be cool if we add more and more popular machine learning tools in dolphinscheduler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Superskyyy commented on issue #9725: [Discussion][Machine Learning] Support AI task and the open source project about MLops
Posted by GitBox <gi...@apache.org>.
Superskyyy commented on issue #9725:
URL: https://github.com/apache/dolphinscheduler/issues/9725#issuecomment-1113644183
Hi, we are planning to initiate our exploration of AIOps at [Apache SkyWalking community](https://github.com/apache/skywalking/discussions). Very interesting to see the discussions here.
Now I'm also looking at Dolphinscheduler to handle our workflow orchestration, and for now, we may as well go in the same direction as yours. I feel like integration with MLFlow package functionalities will be a good point to boost ML-developer experience to the next level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Superskyyy commented on issue #9725: [Discussion][Machine Learning] Support AI task and the open source project about MLops
Posted by GitBox <gi...@apache.org>.
Superskyyy commented on issue #9725:
URL: https://github.com/apache/dolphinscheduler/issues/9725#issuecomment-1119262517
> > Hi, we are planning to initiate our exploration of AIOps at [Apache SkyWalking community](https://github.com/apache/skywalking/discussions). Very interesting to see the discussions here.
>
> >
>
> > Now I'm also looking at Dolphinscheduler to handle our workflow orchestration, and for now, we may as well go in the same direction as yours. I feel like integration with MLFlow package functionalities will be a good point to boost ML-developer experience to the next level.
>
>
>
> Hi, good to see you join the discussion. I just read about your discussion (https://github.com/apache/skywalking/discussions/8883). It should be a great project.
>
>
>
> I think DolphinScheduler will be able to schedule the AIops scenario in the near future. I am enriching its scheduling features in the field of artificial intelligence, and the MVP product is being implemented.
>
>
>
> We can keep talking about that. BTW, I might do some experiments with this data set, but I can't access it right now. https://github.com/CloudWise-OpenSource/GAIA-DataSet
Yeah the dataset repository broke since ？2days ago，if you need I can find a way to send to you. I have a local copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org