You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2019/12/29 01:49:32 UTC

[GitHub] [airflow] xinbinhuang opened a new pull request #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter

xinbinhuang opened a new pull request #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter
URL: https://github.com/apache/airflow/pull/6945
 
 
   
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
     - https://issues.apache.org/jira/browse/AIRFLOW-6318
     
   
   ### Description
   
   - [x] Created from #6860 
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
     1. Subject is separated from body by a blank line
     1. Subject is limited to 50 characters (not including Jira issue reference)
     1. Subject does not end with a period
     1. Subject uses the imperative mood ("add", not "adding")
     1. Body wraps at 72 characters
     1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes how to use it.
     - All the public functions and the classes in the PR contain docstrings that explain what it does
     - If you implement backwards incompatible changes, please leave a note in the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so we can assign it to a appropriate release
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter
URL: https://github.com/apache/airflow/pull/6945#issuecomment-569468023
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6945?src=pr&el=h1) Report
   > Merging [#6945](https://codecov.io/gh/apache/airflow/pull/6945?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7d97d16c47ba181839fd583b8653d2872dc4290b?src=pr&el=desc) will **decrease** coverage by `0.1%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/6945/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6945?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #6945      +/-   ##
   ==========================================
   - Coverage   84.52%   84.42%   -0.11%     
   ==========================================
     Files         680      680              
     Lines       38556    38556              
   ==========================================
   - Hits        32591    32549      -42     
   - Misses       5965     6007      +42
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6945?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: |
   | [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
   | [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
   | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.28% <0%> (+1.17%)` | :arrow_up: |
   | [airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `82.75% <0%> (+6.32%)` | :arrow_up: |
   | ... and [3 more](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6945?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6945?src=pr&el=footer). Last update [7d97d16...fd87800](https://codecov.io/gh/apache/airflow/pull/6945?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter
URL: https://github.com/apache/airflow/pull/6945#discussion_r361846279
 
 

 ##########
 File path: UPDATING.md
 ##########
 @@ -41,6 +41,10 @@ assists users migrating to a new version.
 
 ## Airflow Master
 
+### Change python3 as Dataflow Hooks/Operators default interpreter
+
+Now the `py_interpreter` argument for DataFlow Hooks/Operators has been changed from python2 to python3.
 
 Review comment:
   I think it's good enough for now @mik-laj -> I think what we really need in UPDATING.md is to record all the changes, and before we release 2.0.0 we should prepare some more complete migration guide for the users.
   
   This should include a tool that will mostly automate migrating DAGs and correcting the DAGs they have where needed and print warnings for all cases that need some manual review and corrections (with appropriate instructions). But I think those instructions should be part of that tool rather than UPDATING.md.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter
URL: https://github.com/apache/airflow/pull/6945#issuecomment-569468023
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6945?src=pr&el=h1) Report
   > Merging [#6945](https://codecov.io/gh/apache/airflow/pull/6945?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7d97d16c47ba181839fd583b8653d2872dc4290b?src=pr&el=desc) will **decrease** coverage by `0.1%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/6945/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6945?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #6945      +/-   ##
   ==========================================
   - Coverage   84.52%   84.42%   -0.11%     
   ==========================================
     Files         680      680              
     Lines       38556    38556              
   ==========================================
   - Hits        32591    32549      -42     
   - Misses       5965     6007      +42
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6945?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: |
   | [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
   | [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
   | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.28% <0%> (+1.17%)` | :arrow_up: |
   | [airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `82.75% <0%> (+6.32%)` | :arrow_up: |
   | ... and [3 more](https://codecov.io/gh/apache/airflow/pull/6945/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6945?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6945?src=pr&el=footer). Last update [7d97d16...fd87800](https://codecov.io/gh/apache/airflow/pull/6945?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter
URL: https://github.com/apache/airflow/pull/6945#discussion_r361847106
 
 

 ##########
 File path: UPDATING.md
 ##########
 @@ -41,6 +41,10 @@ assists users migrating to a new version.
 
 ## Airflow Master
 
+### Change python3 as Dataflow Hooks/Operators default interpreter
+
+Now the `py_interpreter` argument for DataFlow Hooks/Operators has been changed from python2 to python3.
 
 Review comment:
   https://issues.apache.org/jira/browse/AIRFLOW-6390 created and proposed that in devlist.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter
URL: https://github.com/apache/airflow/pull/6945#discussion_r361846290
 
 

 ##########
 File path: UPDATING.md
 ##########
 @@ -41,6 +41,10 @@ assists users migrating to a new version.
 
 ## Airflow Master
 
+### Change python3 as Dataflow Hooks/Operators default interpreter
+
+Now the `py_interpreter` argument for DataFlow Hooks/Operators has been changed from python2 to python3.
 
 Review comment:
   I will create an issue about it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] xinbinhuang commented on issue #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter

Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on issue #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter
URL: https://github.com/apache/airflow/pull/6945#issuecomment-569466295
 
 
   @ashb @mik-laj @potiuk . Please take a look and let me know how you think. Thanks!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter

Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter
URL: https://github.com/apache/airflow/pull/6945#discussion_r361844856
 
 

 ##########
 File path: UPDATING.md
 ##########
 @@ -41,6 +41,10 @@ assists users migrating to a new version.
 
 ## Airflow Master
 
+### Change python3 as Dataflow Hooks/Operators default interpreter
+
+Now the `py_interpreter` argument for DataFlow Hooks/Operators has been changed from python2 to python3.
 
 Review comment:
   Can you describe the consequences for the end-user i.e. all jobs is started using Python 3 by default now?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter

Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #6945: [AIRFLOW-6318] Change python3 as Dataflow default intrepreter
URL: https://github.com/apache/airflow/pull/6945
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services