You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/19 11:36:04 UTC
[GitHub] [airflow] darshan-majithiya opened a new pull request #15429: Adding support for Hyperparameter Tuning on Google Cloud AI Platform
darshan-majithiya opened a new pull request #15429:
URL: https://github.com/apache/airflow/pull/15429
The <b>MLEngineStartTrainingJobOperator</b> is modified in <i>airflow/providers/google/cloud/operators/mlengine.py</i> file.
The lines 1192, 1214, 1279, and 1280 are added to the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marcosmarxm commented on a change in pull request #15429: Adding support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
marcosmarxm commented on a change in pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#discussion_r616060692
##########
File path: airflow/providers/google/cloud/operators/mlengine.py
##########
@@ -1189,6 +1189,7 @@ def __init__(
mode: str = 'PRODUCTION',
labels: Optional[Dict[str, str]] = None,
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
+ hyperparameters: Optional[Dict] = None,
Review comment:
Add parameter in class documentation and maybe a link to Google documentation can be helpful to users https://cloud.google.com/ai-platform/training/docs/using-hyperparameter-tuning and https://cloud.google.com/ai-platform/training/docs/reference/rest/v1/projects.jobs#HyperparameterSpec
```
:param hyperparameters Make a dictionary representing your HyperparameterSpec and add it to your training input.
:type Dict
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marcosmarxm commented on a change in pull request #15429: Adding support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
marcosmarxm commented on a change in pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#discussion_r616060692
##########
File path: airflow/providers/google/cloud/operators/mlengine.py
##########
@@ -1189,6 +1189,7 @@ def __init__(
mode: str = 'PRODUCTION',
labels: Optional[Dict[str, str]] = None,
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
+ hyperparameters: Optional[Dict] = None,
Review comment:
Add parameter in class documentation and maybe a link to Google documentation can be helpful to users https://cloud.google.com/ai-platform/training/docs/using-hyperparameter-tuning
```
:param hyperparameters Make a dictionary representing your HyperparameterSpec and add it to your training input.
:type Dict
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #15429: Add support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#discussion_r675248521
##########
File path: airflow/providers/google/cloud/operators/mlengine.py
##########
@@ -1274,6 +1280,9 @@ def execute(self, context):
if self._service_account:
training_request['trainingInput']['serviceAccount'] = self._service_account
+ if self._hyperparameters:
+ training_request['trainingInput']['hyperparameters'] = self._hyperparameters
Review comment:
Can you add tests to avoid regression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] darshan-majithiya commented on a change in pull request #15429: Adding support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
darshan-majithiya commented on a change in pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#discussion_r616609430
##########
File path: airflow/providers/google/cloud/operators/mlengine.py
##########
@@ -1189,6 +1189,7 @@ def __init__(
mode: str = 'PRODUCTION',
labels: Optional[Dict[str, str]] = None,
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
+ hyperparameters: Optional[Dict] = None,
Review comment:
Updated and pushed the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #15429: Adding support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#issuecomment-822398660
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marcosmarxm commented on a change in pull request #15429: Adding support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
marcosmarxm commented on a change in pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#discussion_r616625464
##########
File path: airflow/providers/google/cloud/operators/mlengine.py
##########
@@ -1135,6 +1135,11 @@ class MLEngineStartTrainingJobOperator(BaseOperator):
will be printed out. In 'CLOUD' mode, a real MLEngine training job
creation request will be issued.
:type mode: str
+ :param hyperparameters: Optional HyperparameterSpec dictionary for hyperparameter tuning
+ For further reference, check these:
+ https://cloud.google.com/ai-platform/training/docs/using-hyperparameter-tuning
+ https://cloud.google.com/ai-platform/training/docs/reference/rest/v1/projects.jobs#HyperparameterSpec
+ :type hyperparameters: Dict
Review comment:
Take a look [here](http://rst.ninjs.org/#WW91cnM6Cgo6cGFyYW0gaHlwZXJwYXJhbWV0ZXJzOiBPcHRpb25hbCBIeXBlcnBhcmFtZXRlclNwZWMgZGljdGlvbmFyeSBmb3IgaHlwZXJwYXJhbWV0ZXIgdHVuaW5nCiAgICBGb3IgZnVydGhlciByZWZlcmVuY2UsIGNoZWNrIHRoZXNlOgogICAgaHR0cHM6Ly9jbG91ZC5nb29nbGUuY29tL2FpLXBsYXRmb3JtL3RyYWluaW5nL2RvY3MvdXNpbmctaHlwZXJwYXJhbWV0ZXItdHVuaW5nCiAgICBodHRwczovL2Nsb3VkLmdvb2dsZS5jb20vYWktcGxhdGZvcm0vdHJhaW5pbmcvZG9jcy9yZWZlcmVuY2UvcmVzdC92MS9wcm9qZWN0cy5qb2JzI0h5cGVycGFyYW1ldGVyU3BlYwo6dHlwZSBoeXBlcnBhcmFtZXRlcnM6IERpY3QKCgpBbm90aGVyIHdheSB0byByZXByZXNlbnQgcnN0IGRvY3VtZW50YXRpb246Cgo6cGFyYW0gaHlwZXJwYXJhbWV0ZXJzOiBPcHRpb25hbCBIeXBlcnBhcmFtZXRlclNwZWMgZGljdGlvbmFyeSBmb3IgaHlwZXJwYXJhbWV0ZXIgdHVuaW5nCiAgICBGb3IgZnVydGhlciByZWZlcmVuY2UsIGNoZWNrIHRoZXNlOgogICAgCiAgICAtIGh0dHBzOi8vY2xvdWQuZ29vZ2xlLmNvbS9haS1wbGF0Zm9ybS90cmFpbmluZy9kb2NzL3VzaW5nLWh5cGVycGFyYW1ldGVyLXR1bmluZwogICAgLSBodHRwczovL2Nsb3VkLmdvb2dsZS5jb20vYWktcGxhdGZvcm0vdHJhaW5pbmcvZG9jcy9yZWZlcmVuY2UvcmVzdC92MS9wcm9qZWN0cy5qb2JzI0h5cG
VycGFyYW1ldGVyU3BlYwogICAgLSBgVXNpbmcgSHlwZXJwYXJhbWV0ZXIgdHVuaW5nIDxodHRwczovL2Nsb3VkLmdvb2dsZS5jb20vYWktcGxhdGZvcm0vdHJhaW5pbmcvZG9jcy91c2luZy1oeXBlcnBhcmFtZXRlci10dW5pbmc+YF8KICAgIC0gYEh5cGVycGFyYW1ldGVyU3BlYyBBUEkgZG9jdW1lbnRhdGlvbiA8aHR0cHM6Ly9jbG91ZC5nb29nbGUuY29tL2FpLXBsYXRmb3JtL3RyYWluaW5nL2RvY3MvcmVmZXJlbmNlL3Jlc3QvdjEvcHJvamVjdHMuam9icyNIeXBlcnBhcmFtZXRlclNwZWM+YF8KOnR5cGUgaHlwZXJwYXJhbWV0ZXJzOiBEaWN0Cg==)
I think the way you wrote the documentation they wont be render as lists.
You can test locally using `./breeze build-docs -- --package-filter apache-airflow-providers-google` to generate documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] darshan-majithiya commented on a change in pull request #15429: Add support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
darshan-majithiya commented on a change in pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#discussion_r619843039
##########
File path: airflow/providers/google/cloud/operators/mlengine.py
##########
@@ -1135,6 +1135,11 @@ class MLEngineStartTrainingJobOperator(BaseOperator):
will be printed out. In 'CLOUD' mode, a real MLEngine training job
creation request will be issued.
:type mode: str
+ :param hyperparameters: Optional HyperparameterSpec dictionary for hyperparameter tuning
+ For further reference, check these:
+ https://cloud.google.com/ai-platform/training/docs/using-hyperparameter-tuning
+ https://cloud.google.com/ai-platform/training/docs/reference/rest/v1/projects.jobs#HyperparameterSpec
+ :type hyperparameters: Dict
Review comment:
I'm not able to run pre-commit on my system.
But I've updated and pushed the documentation as per the demo in the link you had shared. Please check it out.
##########
File path: airflow/providers/google/cloud/operators/mlengine.py
##########
@@ -1135,6 +1135,11 @@ class MLEngineStartTrainingJobOperator(BaseOperator):
will be printed out. In 'CLOUD' mode, a real MLEngine training job
creation request will be issued.
:type mode: str
+ :param hyperparameters: Optional HyperparameterSpec dictionary for hyperparameter tuning
+ For further reference, check these:
+ https://cloud.google.com/ai-platform/training/docs/using-hyperparameter-tuning
+ https://cloud.google.com/ai-platform/training/docs/reference/rest/v1/projects.jobs#HyperparameterSpec
+ :type hyperparameters: Dict
Review comment:
I'm not able to run pre-commit on my system. It's throwing permission error. Been trying to find a solution for a while, but haven't figured it yet.
Meanwhile, I've updated and pushed the documentation as per the demo in the link you had shared. Please check it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] darshan-majithiya commented on a change in pull request #15429: Add support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
darshan-majithiya commented on a change in pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#discussion_r617271272
##########
File path: airflow/providers/google/cloud/operators/mlengine.py
##########
@@ -1135,6 +1135,11 @@ class MLEngineStartTrainingJobOperator(BaseOperator):
will be printed out. In 'CLOUD' mode, a real MLEngine training job
creation request will be issued.
:type mode: str
+ :param hyperparameters: Optional HyperparameterSpec dictionary for hyperparameter tuning
+ For further reference, check these:
+ https://cloud.google.com/ai-platform/training/docs/using-hyperparameter-tuning
+ https://cloud.google.com/ai-platform/training/docs/reference/rest/v1/projects.jobs#HyperparameterSpec
+ :type hyperparameters: Dict
Review comment:
I'm not able to run this locally. It exists without any error message.
I tried pushing the code to PR but the build throws an error for the 2nd URL line - https://cloud.google.com/ai-platform/training/docs/reference/rest/v1/projects.jobs#HyperparameterSpec. The line has 112 characters and the max allowed is 110. Will it be okay if I modify the /airflow/pylintrc file and update the max-line-length value to 112? Or is there any other workaround?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] darshan-majithiya commented on a change in pull request #15429: Add support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
darshan-majithiya commented on a change in pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#discussion_r619843039
##########
File path: airflow/providers/google/cloud/operators/mlengine.py
##########
@@ -1135,6 +1135,11 @@ class MLEngineStartTrainingJobOperator(BaseOperator):
will be printed out. In 'CLOUD' mode, a real MLEngine training job
creation request will be issued.
:type mode: str
+ :param hyperparameters: Optional HyperparameterSpec dictionary for hyperparameter tuning
+ For further reference, check these:
+ https://cloud.google.com/ai-platform/training/docs/using-hyperparameter-tuning
+ https://cloud.google.com/ai-platform/training/docs/reference/rest/v1/projects.jobs#HyperparameterSpec
+ :type hyperparameters: Dict
Review comment:
I'm not able to run pre-commit on my system.
But I've updated and pushed the documentation as per the demo in the link you had shared. Please check it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] leahecole commented on pull request #15429: Add support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
leahecole commented on pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#issuecomment-825935494
+1 to the request for test case, and also consider adding an example to https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/example_dags/example_mlengine.py
Thanks for your contribution! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15429: Add support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#issuecomment-885313121
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xinbinhuang commented on pull request #15429: Add support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#issuecomment-824174084
Hi @darshan-majithiya,
Thanks for the PR! Can you also add a test case to the file https://github.com/apache/airflow/blob/master/tests/providers/google/cloud/operators/test_mlengine.py ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #15429: Add support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #15429:
URL: https://github.com/apache/airflow/pull/15429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marcosmarxm commented on a change in pull request #15429: Add support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
marcosmarxm commented on a change in pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#discussion_r617489482
##########
File path: airflow/providers/google/cloud/operators/mlengine.py
##########
@@ -1135,6 +1135,11 @@ class MLEngineStartTrainingJobOperator(BaseOperator):
will be printed out. In 'CLOUD' mode, a real MLEngine training job
creation request will be issued.
:type mode: str
+ :param hyperparameters: Optional HyperparameterSpec dictionary for hyperparameter tuning
+ For further reference, check these:
+ https://cloud.google.com/ai-platform/training/docs/using-hyperparameter-tuning
+ https://cloud.google.com/ai-platform/training/docs/reference/rest/v1/projects.jobs#HyperparameterSpec
+ :type hyperparameters: Dict
Review comment:
Did you enable pre-commit to validate pylint/flake etc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] darshan-majithiya commented on a change in pull request #15429: Add support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
darshan-majithiya commented on a change in pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#discussion_r619843039
##########
File path: airflow/providers/google/cloud/operators/mlengine.py
##########
@@ -1135,6 +1135,11 @@ class MLEngineStartTrainingJobOperator(BaseOperator):
will be printed out. In 'CLOUD' mode, a real MLEngine training job
creation request will be issued.
:type mode: str
+ :param hyperparameters: Optional HyperparameterSpec dictionary for hyperparameter tuning
+ For further reference, check these:
+ https://cloud.google.com/ai-platform/training/docs/using-hyperparameter-tuning
+ https://cloud.google.com/ai-platform/training/docs/reference/rest/v1/projects.jobs#HyperparameterSpec
+ :type hyperparameters: Dict
Review comment:
I'm not able to run pre-commit on my system. It's throwing permission error. Been trying to find a solution for a while, but haven't figured it yet.
Meanwhile, I've updated and pushed the documentation as per the demo in the link you had shared. Please check it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15429: Add support for Hyperparameter Tuning on Google Cloud AI Platform
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15429:
URL: https://github.com/apache/airflow/pull/15429#issuecomment-823819787
[The Workflow run](https://github.com/apache/airflow/actions/runs/769678836) is cancelling this PR. Building images for the PR has failed. Follow the workflow link to check the reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org