You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/24 10:12:22 UTC

[GitHub] [airflow] eladkal opened a new pull request, #25930: [WIP] Deprecate jira provider in favor of atlassian.jira provider

eladkal opened a new pull request, #25930:
URL: https://github.com/apache/airflow/pull/25930

   Related https://github.com/apache/airflow/issues/25669
   **This PR does not change functionality(as requested in the issue) but only names.**
   
   Why I think this name change is better?
   
   1. Jira is Atlassian product and its widely known as such. All atlassian products are under https://www.atlassian.com/
   
   2.  Unlike the previous case of mailing list discussion [Salesforce <-> Tableau](https://lists.apache.org/thread/t9fhhqsdb1x0clg21363s83v27orzk0f) issue https://github.com/apache/airflow/issues/13614 here it makes sense to unite all under 1 provider:
   
       -  Jira is known to be an Atlassian product. Tableau is not known to be a Salesforce product.
       -  There is no specific site for Jira. There is one for Tableau.
       -  Tableau has its own SDK in it's own domain [tableauserverclient](https://github.com/tableau/server-client-python)  while jira SDK is a module in [atlassian-python-api](https://pypi.org/project/atlassian-python-api/)
   
   3.  We will be able to support more Atlassian integration easily without need to have a new provider for each one of them. This is mentioned as motivation  in https://github.com/apache/airflow/issues/25669
   
   PR info:
   1. No system test changes as jira provider doesn't have any
   2. No doc changes as jira provider doesn't have docs other than auto generated ones.
   
   **Things to solve:**
   
   - [ ] static checks will fail on Duplicate integration names found. Not sure how to handle this yet.
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #25930: Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25930:
URL: https://github.com/apache/airflow/pull/25930#issuecomment-1232873556

   LGTM. I like that we have the opportunity to test such rename  on such a small provider with low impact. I wonder what other problems we will find when we release (I am sure we will) - but it looks like a good testing ground simillarly to comon.sql and we wil learn if there are other problems connected 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on a diff in pull request #25930: [WIP] Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #25930:
URL: https://github.com/apache/airflow/pull/25930#discussion_r955851931


##########
setup.py:
##########
@@ -599,12 +599,21 @@ def get_all_db_dependencies() -> List[str]:
 # to separately add providers dependencies - they have been already added as 'providers' extras above
 _all_dependencies = get_unique_dependency_list(EXTRAS_DEPENDENCIES.values())
 
+_all_dependencies_without_airflow_providers = list(
+    filter(lambda k: 'apache-airflow-' not in k, _all_dependencies)
+)
+
 # All user extras here
-EXTRAS_DEPENDENCIES["all"] = _all_dependencies
+# all is purely development extra and it should contain only direct dependencies of Airflow
+# It should contain all dependencies of airflow and dependencies of all community providers,
+# but not the providers themselves
+EXTRAS_DEPENDENCIES["all"] = _all_dependencies_without_airflow_providers
 
 # This can be simplified to devel_hadoop + _all_dependencies due to inclusions
 # but we keep it for explicit sake. We are de-duplicating it anyway.
-devel_all = get_unique_dependency_list([_all_dependencies, doc, devel, devel_hadoop])
+devel_all = get_unique_dependency_list(
+    [_all_dependencies_without_airflow_providers, doc, devel, devel_hadoop]
+)

Review Comment:
   @potiuk please double check me on the changes to `setup.py`
   I had to change `devel_all` as well to make the CI recognize the new provider



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #25930: Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
potiuk merged PR #25930:
URL: https://github.com/apache/airflow/pull/25930


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #25930: Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25930:
URL: https://github.com/apache/airflow/pull/25930#issuecomment-1232913097

   We can also add "common.taskflow" as described in https://github.com/apache/airflow/issues/26016#issuecomment-1231698030


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #25930: Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25930:
URL: https://github.com/apache/airflow/pull/25930#issuecomment-1231489975

   Now after #26302 is merged  you need to rebase and regenerate the imges rather than resolve the conflicts one more time :) . Shoudl be the last one :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #25930: Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25930:
URL: https://github.com/apache/airflow/pull/25930#issuecomment-1232913280

   cc: @uranusjr 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #25930: Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25930:
URL: https://github.com/apache/airflow/pull/25930#issuecomment-1231512332

   BTW: @eladkal - after the #26032 merged the output will be far more readable and you will know more what is going on:
   
   Example after solving some of my conflicts in #25678 
   
   ![image](https://user-images.githubusercontent.com/595491/187421269-7a7ad1cf-d33b-46b1-8a85-58ef2a2442f7.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #25930: Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25930:
URL: https://github.com/apache/airflow/pull/25930#issuecomment-1231474880

   We have also the comment in the hash image:
   
   > This file is automatically generated by pre-commit. If you have a conflict with this file
   Please do not solve it but run `breeze setup regenerate-command-images`.
   This command should fix the conflict and regenerate help images that you have conflict with.
   
   But I understand how this could get wrong so #26032 will make it far more consistent and automated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on pull request #25930: Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #25930:
URL: https://github.com/apache/airflow/pull/25930#issuecomment-1231450295

   @potiuk there is something strange with pre-commit.
   It passes locally but in CI it produce weird error - it looks like something is broken?
   https://github.com/apache/airflow/runs/8088945322?check_suite_focus=true#step:10:1473
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on pull request #25930: Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #25930:
URL: https://github.com/apache/airflow/pull/25930#issuecomment-1232886293

   yeah extracting time/date related operators from core to common.time (or some other name) is also on my todo list


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on pull request #25930: [WIP] Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #25930:
URL: https://github.com/apache/airflow/pull/25930#issuecomment-1225847743

   @potiuk could use your help to find the issue with:
   ```
     #55 17.81 ERROR: Could not find a version that satisfies the requirement apache-airflow-providers-atlassian-jira (from apache-airflow[devel-ci]) (from versions: none)
     #55 17.81 ERROR: No matching distribution found for apache-airflow-providers-atlassian-jira
   ```
   
   from what I see `devl-ci` providers are taking from `generated/provider_dependencies.json`
   so any clue why it's missing?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on a diff in pull request #25930: Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #25930:
URL: https://github.com/apache/airflow/pull/25930#discussion_r956038963


##########
setup.py:
##########
@@ -599,12 +599,21 @@ def get_all_db_dependencies() -> List[str]:
 # to separately add providers dependencies - they have been already added as 'providers' extras above
 _all_dependencies = get_unique_dependency_list(EXTRAS_DEPENDENCIES.values())
 
+_all_dependencies_without_airflow_providers = list(
+    filter(lambda k: 'apache-airflow-' not in k, _all_dependencies)
+)
+
 # All user extras here
-EXTRAS_DEPENDENCIES["all"] = _all_dependencies
+# all is purely development extra and it should contain only direct dependencies of Airflow
+# It should contain all dependencies of airflow and dependencies of all community providers,
+# but not the providers themselves
+EXTRAS_DEPENDENCIES["all"] = _all_dependencies_without_airflow_providers
 
 # This can be simplified to devel_hadoop + _all_dependencies due to inclusions
 # but we keep it for explicit sake. We are de-duplicating it anyway.
-devel_all = get_unique_dependency_list([_all_dependencies, doc, devel, devel_hadoop])
+devel_all = get_unique_dependency_list(
+    [_all_dependencies_without_airflow_providers, doc, devel, devel_hadoop]
+)

Review Comment:
   Great. Everything should be green now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on a diff in pull request #25930: [WIP] Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #25930:
URL: https://github.com/apache/airflow/pull/25930#discussion_r956023277


##########
setup.py:
##########
@@ -599,12 +599,21 @@ def get_all_db_dependencies() -> List[str]:
 # to separately add providers dependencies - they have been already added as 'providers' extras above
 _all_dependencies = get_unique_dependency_list(EXTRAS_DEPENDENCIES.values())
 
+_all_dependencies_without_airflow_providers = list(
+    filter(lambda k: 'apache-airflow-' not in k, _all_dependencies)
+)
+
 # All user extras here
-EXTRAS_DEPENDENCIES["all"] = _all_dependencies
+# all is purely development extra and it should contain only direct dependencies of Airflow
+# It should contain all dependencies of airflow and dependencies of all community providers,
+# but not the providers themselves
+EXTRAS_DEPENDENCIES["all"] = _all_dependencies_without_airflow_providers
 
 # This can be simplified to devel_hadoop + _all_dependencies due to inclusions
 # but we keep it for explicit sake. We are de-duplicating it anyway.
-devel_all = get_unique_dependency_list([_all_dependencies, doc, devel, devel_hadoop])
+devel_all = get_unique_dependency_list(
+    [_all_dependencies_without_airflow_providers, doc, devel, devel_hadoop]
+)

Review Comment:
   Yeah. That's right :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on pull request #25930: Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #25930:
URL: https://github.com/apache/airflow/pull/25930#issuecomment-1232848444

   @potiuk finally green build :) 
   can you please re-review? there were changes since your approval


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #25930: Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25930:
URL: https://github.com/apache/airflow/pull/25930#issuecomment-1231471626

   From: #26032 
   
   > This change also introduces much more selective image generation
   which will make the image generation much faster in case you
   change any of breeze code. Previously, any change in any breeze
   command parameters triggered regeneration of the images.
   With this change, only those commands that get changed get
   regenerated (and information about which images need to be regenerated
   is printed).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #25930: Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25930:
URL: https://github.com/apache/airflow/pull/25930#issuecomment-1231470690

   If you resolved the conflict manually - this might happen. 
   
   Actually: https://github.com/apache/airflow/pull/26032 is supposed to handle it much better (needs approval).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #25930: Deprecate jira provider in favor of atlassian.jira provider

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25930:
URL: https://github.com/apache/airflow/pull/25930#issuecomment-1231469123

   Quote from the output:
   
   ```
    If you see that  images got re-generated in CI, please run:
   
       breeze setup regenerate-command-images
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org