You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/02/22 19:28:21 UTC

[GitHub] [airflow] BasPH opened a new pull request #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result

BasPH opened a new pull request #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508
 
 
   The SageMakerTrainingOperator returns the response of `hook.describe_training_job()` wrapped in a dict with key `Training`. I don't see it used anywhere in the codebase, nor anywhere on the internet, so I suggest to simplify the result and return the JSON returned by `hook.describe_training_job()`.
   
   Before:
   
   ```python
   {
       'Training': {... SageMaker result here ...}
   }
   ```
   
   After:
   ```python
   {... SageMaker result here ...}
   ```
   
   This way, you avoid one useless step processing the results:
   
   ```python
   # before
   jobname = result["Training"]["TrainingJobName"]
   # after
   jobname = result["TrainingJobName"]
   ```
   
   ---
   Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   <sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508#issuecomment-590295099
 
 
   The reason is for consistency with the rest of the suite of Sagemaker operators:
   
   > I guess the CR author is only using SageMaker training operator. So this ‘Training’ field is weird and unnecessary to him. But if you look at other SageMaker operators, for example the batch transform one (https://github.com/apache/airflow/blob/1a9d218611502118132a93b0c12c1ad511cbdd6e/airflow/providers/amazon/aws/operators/sagemaker_transform.py#L117-L123) , it has two description results. So we definitely need a ‘Model’ field and a ‘Transform’ field here. And to make all SageMaker operators being consistent in the result format, I just had that ‘Training’ field although there’s only one description result.
   >
   > In this case I personally value consistency over cleanness (otherwise I won’t write it that way). So I don’t recommend to remove this ‘Training’ key although it’s useless. However if you guys think it’s better to remove it according to Airflow culture. I totally respect it. Just let Yuan Zha [Ed: member of the Sagemaker team at AWS, I have his email] knows this. He has more information on the impact of this change (whether customer’s workflow will be broken) and can stand for Amazon better than I can.
   
   What do you think @BasPH? I think I'm leaning to a -0 after this clarification.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508#issuecomment-590296475
 
 
   > The reason is for consistency with the rest of the suite of Sagemaker operators:
   > 
   > > I guess the CR author is only using SageMaker training operator. So this ‘Training’ field is weird and unnecessary to him. But if you look at other SageMaker operators, for example the batch transform one (https://github.com/apache/airflow/blob/1a9d218611502118132a93b0c12c1ad511cbdd6e/airflow/providers/amazon/aws/operators/sagemaker_transform.py#L117-L123
   > > ) , it has two description results. So we definitely need a ‘Model’ field and a ‘Transform’ field here. And to make all SageMaker operators being consistent in the result format, I just had that ‘Training’ field although there’s only one description result.
   > > In this case I personally value consistency over cleanness (otherwise I won’t write it that way). So I don’t recommend to remove this ‘Training’ key although it’s useless. However if you guys think it’s better to remove it according to Airflow culture. I totally respect it. Just let Yuan Zha [Ed: member of the Sagemaker team at AWS, I have his email] knows this. He has more information on the impact of this change (whether customer’s workflow will be broken) and can stand for Amazon better than I can.
   > 
   > What do you think @BasPH? I think I'm leaning to a -0 after this clarification.
   
   We we plan to revert the change in this PR, we should change this PR to add "comments" above the response so that the future users know why it is here

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] stale[bot] closed pull request #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result

Posted by GitBox <gi...@apache.org>.
stale[bot] closed pull request #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508#issuecomment-590001224
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=h1) Report
   > Merging [#7508](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/61a8bb65818521ccbb846e647103535b3e36b26d?src=pr&el=desc) will **decrease** coverage by `0.28%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7508/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7508      +/-   ##
   ==========================================
   - Coverage   86.82%   86.54%   -0.29%     
   ==========================================
     Files         891      891              
     Lines       42095    42095              
   ==========================================
   - Hits        36549    36431     -118     
   - Misses       5546     5664     +118
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...oviders/amazon/aws/operators/sagemaker\_training.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvc2FnZW1ha2VyX3RyYWluaW5nLnB5) | `100% <100%> (ø)` | :arrow_up: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: |
   | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.38% <0%> (-25.52%)` | :arrow_down: |
   | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.12% <0%> (+0.19%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=footer). Last update [61a8bb6...9b0ec04](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] stale[bot] commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result

Posted by GitBox <gi...@apache.org>.
stale[bot] commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508#issuecomment-612418774
 
 
   This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508#issuecomment-590001224
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=h1) Report
   > Merging [#7508](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/61a8bb65818521ccbb846e647103535b3e36b26d?src=pr&el=desc) will **decrease** coverage by `0.37%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7508/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7508      +/-   ##
   ==========================================
   - Coverage   86.82%   86.45%   -0.38%     
   ==========================================
     Files         891      891              
     Lines       42095    42095              
   ==========================================
   - Hits        36549    36392     -157     
   - Misses       5546     5703     +157
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...oviders/amazon/aws/operators/sagemaker\_training.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvc2FnZW1ha2VyX3RyYWluaW5nLnB5) | `100% <100%> (ø)` | :arrow_up: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `50% <0%> (-50%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: |
   | [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `52.94% <0%> (-32.36%)` | :arrow_down: |
   | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.38% <0%> (-25.52%)` | :arrow_down: |
   | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `77.46% <0%> (-16.91%)` | :arrow_down: |
   | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `90.08% <0%> (-1.66%)` | :arrow_down: |
   | ... and [2 more](https://codecov.io/gh/apache/airflow/pull/7508/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=footer). Last update [61a8bb6...04bbe56](https://codecov.io/gh/apache/airflow/pull/7508?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508#issuecomment-590062760
 
 
   Can't see an obvious reason for this, I'll ping Yang via email too in case he doesn't see the GitHub notification

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] BasPH commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result

Posted by GitBox <gi...@apache.org>.
BasPH commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508#issuecomment-590297481
 
 
   Makes sense, let's leave it. I can update this PR to undo the changes and simply add a comment with this explanation.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services