You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "mathewjacob1002 (via GitHub)" <gi...@apache.org> on 2023/07/12 19:54:15 UTC
[GitHub] [spark] mathewjacob1002 opened a new pull request, #41968: [Spark Ticket Here][WIP] Clean Up Deepspeed Code
mathewjacob1002 opened a new pull request, #41968:
URL: https://github.com/apache/spark/pull/41968
Follow up on @HyukjinKwon comments on [this pr](https://github.com/apache/spark/pull/41770).
###Checklist
- [ ] Add docs on Deepspeed Distributor
- [ ] Make all variable names lowercase in test_deepspeed_distributor per PEP8
- [ ] Make docstrings adhere to numpy standard
- [ ] Add an example of how to use deepspeed distributor in the examples section(same as TorchDistributor basically)
- [ ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mathewjacob1002 commented on pull request #41968: [Spark Ticket Here][WIP] Clean Up Deepspeed Code
Posted by "mathewjacob1002 (via GitHub)" <gi...@apache.org>.
mathewjacob1002 commented on PR #41968:
URL: https://github.com/apache/spark/pull/41968#issuecomment-1633257981
@HyukjinKwon do you think we should add the examples only after we get everything in DeepspeedTorchDistributor feature-complete?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #41968: [SPARK-44264]Clean Up Deepspeed Code
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #41968:
URL: https://github.com/apache/spark/pull/41968#issuecomment-1633328141
Ah just read
> do you think we should add the examples only after we get everything in DeepspeedTorchDistributor feature-complete?
Yes, that works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #41968: [SPARK-44264][PYTHON][ML][FOLLOW-UP] Clean Up Deepspeed Code
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #41968: [SPARK-44264][PYTHON][ML][FOLLOW-UP] Clean Up Deepspeed Code
URL: https://github.com/apache/spark/pull/41968
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mathewjacob1002 commented on pull request #41968: [SPARK-44264][PYTHON][ML][FOLLOW-UP] Clean Up Deepspeed Code
Posted by "mathewjacob1002 (via GitHub)" <gi...@apache.org>.
mathewjacob1002 commented on PR #41968:
URL: https://github.com/apache/spark/pull/41968#issuecomment-1636671632
@HyukjinKwon Everything should work now :D could we merge this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mathewjacob1002 commented on a diff in pull request #41968: [SPARK-44264][PYTHON][ML][FOLLOW-UP] Clean Up Deepspeed Code
Posted by "mathewjacob1002 (via GitHub)" <gi...@apache.org>.
mathewjacob1002 commented on code in PR #41968:
URL: https://github.com/apache/spark/pull/41968#discussion_r1261879148
##########
python/docs/source/reference/pyspark.ml.rst:
##########
@@ -351,6 +351,7 @@ Distributor
:toctree: api/
TorchDistributor
+ DeepspeedTorchDistributor
Review Comment:
oh it's torch.deepspeed.DeepspeedTorchDistributor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41968: [SPARK-44264][PYTHON][ML][FOLLOW-UP] Clean Up Deepspeed Code
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41968:
URL: https://github.com/apache/spark/pull/41968#discussion_r1261876509
##########
python/docs/source/reference/pyspark.ml.rst:
##########
@@ -351,6 +351,7 @@ Distributor
:toctree: api/
TorchDistributor
+ DeepspeedTorchDistributor
Review Comment:
Seems like it fails:
```
2023-07-12T23:31:59.4320534Z Warning, treated as error:
2023-07-12T23:31:59.4322346Z [autosummary] failed to import 'pyspark.ml.torch.distributor.DeepspeedTorchDistributor': no module named pyspark.ml.torch.distributor.DeepspeedTorchDistributor
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #41968: [SPARK-44264][PYTHON][ML][FOLLOW-UP] Clean Up Deepspeed Code
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #41968:
URL: https://github.com/apache/spark/pull/41968#issuecomment-1637220928
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org