You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/02/22 19:28:21 UTC
[GitHub] [airflow] BasPH opened a new pull request #7508: [AIRFLOW-6886]
Cleanup the SageMakerTrainingOperator result
BasPH opened a new pull request #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508
The SageMakerTrainingOperator returns the response of `hook.describe_training_job()` wrapped in a dict with key `Training`. I don't see it used anywhere in the codebase, nor anywhere on the internet, so I suggest to simplify the result and return the JSON returned by `hook.describe_training_job()`.
Before:
```python
{
'Training': {... SageMaker result here ...}
}
```
After:
```python
{... SageMaker result here ...}
```
This way, you avoid one useless step processing the results:
```python
# before
jobname = result["Training"]["TrainingJobName"]
# after
jobname = result["TrainingJobName"]
```
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
<sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #7508: [AIRFLOW-6886] Cleanup the
SageMakerTrainingOperator result
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508#issuecomment-590295099
The reason is for consistency with the rest of the suite of Sagemaker operators:
> I guess the CR author is only using SageMaker training operator. So this ‘Training’ field is weird and unnecessary to him. But if you look at other SageMaker operators, for example the batch transform one (https://github.com/apache/airflow/blob/1a9d218611502118132a93b0c12c1ad511cbdd6e/airflow/providers/amazon/aws/operators/sagemaker_transform.py#L117-L123) , it has two description results. So we definitely need a ‘Model’ field and a ‘Transform’ field here. And to make all SageMaker operators being consistent in the result format, I just had that ‘Training’ field although there’s only one description result.
>
> In this case I personally value consistency over cleanness (otherwise I won’t write it that way). So I don’t recommend to remove this ‘Training’ key although it’s useless. However if you guys think it’s better to remove it according to Airflow culture. I totally respect it. Just let Yuan Zha [Ed: member of the Sagemaker team at AWS, I have his email] knows this. He has more information on the impact of this change (whether customer’s workflow will be broken) and can stand for Amazon better than I can.
What do you think @BasPH? I think I'm leaning to a -0 after this clarification.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on issue #7508: [AIRFLOW-6886] Cleanup
the SageMakerTrainingOperator result
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508#issuecomment-590296475
> The reason is for consistency with the rest of the suite of Sagemaker operators:
>
> > I guess the CR author is only using SageMaker training operator. So this ‘Training’ field is weird and unnecessary to him. But if you look at other SageMaker operators, for example the batch transform one (https://github.com/apache/airflow/blob/1a9d218611502118132a93b0c12c1ad511cbdd6e/airflow/providers/amazon/aws/operators/sagemaker_transform.py#L117-L123
> > ) , it has two description results. So we definitely need a ‘Model’ field and a ‘Transform’ field here. And to make all SageMaker operators being consistent in the result format, I just had that ‘Training’ field although there’s only one description result.
> > In this case I personally value consistency over cleanness (otherwise I won’t write it that way). So I don’t recommend to remove this ‘Training’ key although it’s useless. However if you guys think it’s better to remove it according to Airflow culture. I totally respect it. Just let Yuan Zha [Ed: member of the Sagemaker team at AWS, I have his email] knows this. He has more information on the impact of this change (whether customer’s workflow will be broken) and can stand for Amazon better than I can.
>
> What do you think @BasPH? I think I'm leaning to a -0 after this clarification.
We we plan to revert the change in this PR, we should change this PR to add "comments" above the response so that the future users know why it is here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] stale[bot] closed pull request #7508: [AIRFLOW-6886]
Cleanup the SageMakerTrainingOperator result
Posted by GitBox <gi...@apache.org>.
stale[bot] closed pull request #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7508: [AIRFLOW-6886]
Cleanup the SageMakerTrainingOperator result
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508#issuecomment-590001224
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=h1) Report
> Merging [#7508](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/61a8bb65818521ccbb846e647103535b3e36b26d?src=pr&el=desc) will **decrease** coverage by `0.28%`.
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7508 +/- ##
==========================================
- Coverage 86.82% 86.54% -0.29%
==========================================
Files 891 891
Lines 42095 42095
==========================================
- Hits 36549 36431 -118
- Misses 5546 5664 +118
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...oviders/amazon/aws/operators/sagemaker\_training.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvc2FnZW1ha2VyX3RyYWluaW5nLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.38% <0%> (-25.52%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.12% <0%> (+0.19%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=footer). Last update [61a8bb6...9b0ec04](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #7508: [AIRFLOW-6886]
Cleanup the SageMakerTrainingOperator result
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508#issuecomment-612418774
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7508:
[AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508#issuecomment-590001224
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=h1) Report
> Merging [#7508](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/61a8bb65818521ccbb846e647103535b3e36b26d?src=pr&el=desc) will **decrease** coverage by `0.37%`.
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7508 +/- ##
==========================================
- Coverage 86.82% 86.45% -0.38%
==========================================
Files 891 891
Lines 42095 42095
==========================================
- Hits 36549 36392 -157
- Misses 5546 5703 +157
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...oviders/amazon/aws/operators/sagemaker\_training.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvc2FnZW1ha2VyX3RyYWluaW5nLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `50% <0%> (-50%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: |
| [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `52.94% <0%> (-32.36%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.38% <0%> (-25.52%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `77.46% <0%> (-16.91%)` | :arrow_down: |
| [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `90.08% <0%> (-1.66%)` | :arrow_down: |
| ... and [2 more](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=footer). Last update [61a8bb6...04bbe56](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #7508: [AIRFLOW-6886] Cleanup the
SageMakerTrainingOperator result
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508#issuecomment-590062760
Can't see an obvious reason for this, I'll ping Yang via email too in case he doesn't see the GitHub notification
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] BasPH commented on issue #7508: [AIRFLOW-6886] Cleanup
the SageMakerTrainingOperator result
Posted by GitBox <gi...@apache.org>.
BasPH commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508#issuecomment-590297481
Makes sense, let's leave it. I can update this PR to undo the changes and simply add a comment with this explanation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services