You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/09 21:42:01 UTC

[GitHub] [airflow] dstandish opened a new pull request #20184: Add and use `exactly_one` helper

dstandish opened a new pull request #20184:
URL: https://github.com/apache/airflow/pull/20184


   Bitwise XOR is not very readable, only works with True / False, and only works with two values.  E.g. if you need to test "exactly one of a, b, or c", you cannot do  a ^ b ^ c.  I add an "exactly_one" boolean helper to address all of these shortcomings.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#issuecomment-991237436


   That tread confirms the old Chinese wisdom: "the smaller the PR, the longer conversation" :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#issuecomment-990312239


   utils.helpers?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766764509



##########
File path: airflow/providers/amazon/aws/operators/emr_add_steps.py
##########
@@ -62,7 +63,7 @@ def __init__(
     ):
         if kwargs.get('xcom_push') is not None:
             raise AirflowException("'xcom_push' was deprecated, use 'do_xcom_push' instead")
-        if not (job_flow_id is None) ^ (job_flow_name is None):
+        if not exactly_one(job_flow_id is not None, job_flow_name is not None):

Review comment:
       yup true forgot about that thanks




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766203082



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """

Review comment:
       But let me be honest - I cheated: https://stackoverflow.com/questions/22798938/how-to-check-if-exactly-one-value-in-an-iterable-resolves-to-true




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766195013



##########
File path: airflow/utils/boolean.py
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       > BTW. @dstandish - you do realise that a ^ b ^ c is (one_or_three() ) :) ?
   
   yeah it's "exactly one OR all three" -- precisely the problem that my very old-fashioned and not-so-new-fangled code addresses :) 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#issuecomment-990310365


   I searched for a suitable spot but did not find.  You have a suggestion?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766207478



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       exploring that thought...
   
   supposing we parameterize the map function like so:
   
   ```python
   def exactly_one(*args, func=bool):
       """
       Returns True if exactly one of *args is "truthy", and False otherwise.
       :param args:
       :return:
       """
       return sum(map(func, args)) == 1
   ```
   
   then in your case you could do
   
   ```python
   exactly_one(a, b, c, func=lambda x: x != NOTSET)
   ```
   
   But given that you could achieve the same result, with the simpler function in this way:
   
   ```python
   exactly_one(*map(lambda x: x !=NOTSET, vals))
   ```
   
   I think maybe keeping the function simpler is better?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766194040



##########
File path: airflow/utils/boolean.py
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       BTW. @dstandish - you do realise that a ^ b ^ c  is (one_or_three() ) :) ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766199550



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """

Review comment:
       ok tests pass you win the prize.
   
   nothing like a little code golf now and  again




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766197880



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """

Review comment:
       Challenge accepted:
   
   How about
   
   `return sum(map(bool,args) == 1`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766207478



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       But to explore that thought...
   
   supposing we parameterize the map function like so:
   
   ```python
   def exactly_one(*args, func=bool):
       """
       Returns True if exactly one of *args is "truthy", and False otherwise.
       :param args:
       :return:
       """
       return sum(map(func, args)) == 1
   ```
   
   then in your case you could do
   
   ```python
   exactly_one(a, b, c, func=lambda x: x != NOTSET)
   ```
   
   But given that you could achieve the same result, with the simpler function in this way:
   
   ```python
   exactly_one(*map(lambda x: x !=NOTSET, vals))
   ```
   
   I think maybe keeping the function simpler is better?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766205394



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       so the meaning is, "exactly one is truthy", not "exactly one is set"




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#issuecomment-991237436


   That tread confirms the old Chinese wisdom: "the smaller the PR, the longer conversation" :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766198588



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """

Review comment:
       we might have a winner




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766204262



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       My intention is that the user must provide args that are truthy/falsy  
   
   So in that case you would do 
   
   ```python
   exactly_one(a != NOTSET, b != NOTSET)
   
   ```
   
   etc.... what do you think?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] github-actions[bot] commented on pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#issuecomment-993983874


   The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766196230



##########
File path: airflow/utils/boolean.py
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       like in the 3 case you can do `(a ^ b ^ c) and not (a and b and c)`...
   which suggests maybe `return functools.reduce(operator.xor, map(bool, args)) and not functools.reduce(operator.and_, map(bool, args))` would work
   however, this is not generally true -- indeed it fails when you have 4 elements!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766196230



##########
File path: airflow/utils/boolean.py
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       like in the 3 case you can do `(a ^ b ^ c) and not (a and b and c)`... however, this is not generally true -- indeed it fails when you have 4 elements!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766235448



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       > I went with `exactly_one(execution_date, run_id)` because AFAIK those values obey truthy evaluation. Are you suggesting that we do `is not None` because we should never do truthy evaluation? Or because in this particular case there is a reason not to?
   
   Because `run_id` can be an empty string, and we want to show a different error for this situation.
   
   > can you clarify?
   
   Basically the above. The main purpose of this function (well, the xor checks it is replacing) is to check whether arguments are passed, not whether those arguments are falsy. And if you don’t either explicitly use `is not None`, or include stricter evaluation logic in `exactly_one`, this creates a subtle semantic difference between the caller site and the body of `exactly_one` that can potentially trip people up and cause subtle, unintended behavioural change. And this situation is complete avoidable by changing the evaluation logic to not rely on falsiness.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766185727



##########
File path: airflow/utils/boolean.py
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       ```suggestion
       return functools.reduce(operator.xor, map(bool, args))
   ```
   
   😁 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766230039



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       >  That would work. In that case though, I’d make some of the existing changes more explicit e.g. exactly_one(execution_date is not None, run_id is not None)
   
   I went with `exactly_one(execution_date, run_id)` because AFAIK those values obey truthy evaluation.  Are you suggesting that we do `is not None` because we should never do truthy evaluation?  Or because in this particular case there is a reason not to?  When it was XOR there was a reason (namely because XOR doesn't handle truthy), but seems here it should be OK,  and as you pointed out it's less verbose -- wdyt?
   
   > that’s a bit verbose to me though, so parametrising exactly_one should still be considered IMO
   
   I did explore the idea above, and arrived at the feeling that it wasn't worth it.  But if you feel strongly we should parameterize please say so and perhaps propose a manner.  I'm not strongly against parameterizing, it just makes it a little more complicated to understand the behavior and to me it seemed not worth it (since it doesn't really make usage more elegant than using map and a lambda).
   
   > On second thought, this falsy-none problem may be an unnecessary pitfall in the first place, indicating we should build safeguards in exactly_one in the first place.
   
   can you clarify?
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766202344



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       This can be problematic if the argument is of a type that can accept a falsy value, or if the default is not falsy (e.g. `NOTSET`). This should Can accept an optional `check_default=` keyword argument and check for default-ness with that instead (the default would be e.g. `functools.partial(operator.is_not, None)`).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766202344



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       This can be problematic if the argument is of a type that can accept a falsy value, or if the default is not falsy (e.g. `NOTSET`). This should Can accept an optional `sentinel=None` keyword argument and check for default-ness with `is` instead.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766188190



##########
File path: airflow/utils/boolean.py
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       lol




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766197880



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """

Review comment:
       Challenge accepted:
   
   How about
   
   `return sum(map(bool,args)) == 1`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766201223



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """

Review comment:
       Though it's less efficient in certain cases :) 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766202298



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """

Review comment:
       yeah mine will fail faster but yours is cleaner 🤷




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766192901



##########
File path: airflow/utils/boolean.py
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       Oh yeah fair point - xor trick doesn't work for more than two args.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766219374



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       That would work. In that case though, I’d make some of the existing changes more explicit e.g. `exactly_one(execution_date is not None, run_id is not None)` (that’s a bit verbose to me though, so parametrising `exactly_one` should still be considered IMO), and document the function in detail so future contributors don’t fall in this pitfall. On second thought, this falsy-none problem may be an unnecessary pitfall in the first place, indicating we should build safeguards in `exactly_one` in the first place.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish merged pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish merged pull request #20184:
URL: https://github.com/apache/airflow/pull/20184


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#issuecomment-990315017


   > utils.helpers?
   
   hah didn't realize it was such a grab bag in there looks like it will fit in well
   
   i thought i recalled that being mostly task-related helpers but does not appear to be the case!
   
   thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766191797



##########
File path: airflow/utils/boolean.py
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       hmm so the that's the thing with that XOR, this fails:
   ```
   >>> functools.reduce(operator.xor, map(bool, ['a', 'a', 'a']))
   True
   ```
   
   (same goes for True in place of 'a')
   
   The issue is that `True ^ True ^ True is True`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766190108



##########
File path: airflow/utils/boolean.py
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       Nice. And ... it can be parallelised ;) 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766230039



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       >  That would work. In that case though, I’d make some of the existing changes more explicit e.g. exactly_one(execution_date is not None, run_id is not None)
   
   I went with `exactly_one(execution_date, run_id)` because AFAIK those values obey truthy evaluation.  Are you suggesting that we do `is not None` because we should never do truthy evaluation?  Or because in this particular case there is a reason not to?  When it was XOR there was a reason (namely because XOR doesn't handle truthy), but seems here it should be OK,  and as you pointed out it's less verbose -- wdyt?
   
   > that’s a bit verbose to me though, so parametrising exactly_one should still be considered IMO
   
   I did explore the idea above, and arrived at the feeling that it wasn't worth it.  But if you feel strongly we should parameterize please say so and perhaps propose a manner.  It doesn't really cost too much to parameterize just makes it a little more complicated to understand the behavior.
   
   > On second thought, this falsy-none problem may be an unnecessary pitfall in the first place, indicating we should build safeguards in exactly_one in the first place.
   
   can you clarify?
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dstandish commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766207478



##########
File path: airflow/utils/helpers.py
##########
@@ -244,3 +244,18 @@ def build_airflow_url_with_query(query: Dict[str, Any]) -> str:
     view = conf.get('webserver', 'dag_default_view').lower()
     url = url_for(f"Airflow.{view}")
     return f"{url}?{parse.urlencode(query)}"
+
+
+def exactly_one(*args):
+    """
+    Returns True if exactly one of *args is "truthy", and False otherwise.
+    :param args:
+    :return:
+    """
+    has_one = False
+    for arg in args:
+        if arg and has_one:
+            return False
+        elif arg:
+            has_one = True
+    return has_one

Review comment:
       exploring that thought...
   
   ```python
   def exactly_one(*args, func=bool):
       """
       Returns True if exactly one of *args is "truthy", and False otherwise.
       :param args:
       :return:
       """
       return sum(map(func, args)) == 1
   
   exactly_one(a, b, c, func=lambda x: x != NOTSET)
   ```
   
   Given that you could achieve the same result, with the simpler function in this way:
   
   ```python
   exactly_one(*map(lambda x: x !=NOTSET, vals))
   ```
   
   I think maybe keeping the function simpler is better?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #20184: Add and use `exactly_one` helper

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #20184:
URL: https://github.com/apache/airflow/pull/20184#discussion_r766515023



##########
File path: airflow/providers/amazon/aws/operators/emr_add_steps.py
##########
@@ -62,7 +63,7 @@ def __init__(
     ):
         if kwargs.get('xcom_push') is not None:
             raise AirflowException("'xcom_push' was deprecated, use 'do_xcom_push' instead")
-        if not (job_flow_id is None) ^ (job_flow_name is None):
+        if not exactly_one(job_flow_id is not None, job_flow_name is not None):

Review comment:
       We can't use this in providers -- otherwise it means the provider will only work with Airflow 2.3+, and while it's possible we can do that, this change isn't worth imposing that limit..




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org