[GitHub] [airflow] nivangio opened a new issue, #29931: Type hinting in `DatabricksSubmitRunOperator` is not compatible with parameter passing possibilities

["nivangio (via GitHub)" <gi...@apache.org>]

nivangio opened a new issue, #29931:
URL: https://github.com/apache/airflow/issues/29931

   ### Apache Airflow Provider(s)
   
   databricks
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-databricks==4.0.0
   
   ### Apache Airflow version
   
   2.4.3
   
   ### Operating System
   
   MAC OS Monterrey
   
   ### Deployment
   
   Virtualenv installation
   
   ### Deployment details
   
   The issue is consistent across multiple Airflow deployments (locally on Docker Compose, remotely on MWAA in AWS, locally using virualenv)
   
   ### What happened
   
   This Issue is the continuation of the discussion started here: https://github.com/apache/airflow/issues/29746
   
   
   `base_parameters` parameter in `DatabricksSubmitRunOperator` is typed as `dict[str,str]` when it actually accepts a broader variety of types in the value section, also supported by `normalise_json_content`.
   
   This mistyping causes a contraidctory behaviour where `DatabricksSubmitRunOperator` runs succesfully and as expected but mypy checks fail.
   
   ### What you think should happen instead
   
   `DatabricksSubmitRunOperator` should definitely accept a broader type variety. 
   
   Given that `normalise_json_content` is a recursive function, all parameters in `DatabricksSubmitRunOperator` should be of recursable type and probably the type hinting of the function itself needs a change (actually its `content` parameter is untyped ATM) something in the direction of:
   
   ```python3
   DatabricksSubmitRunOperatorParameter: TypeAlias = Union[bool, int, float, str, Iterable['DatabricksSubmitRunOperatorParameter']]
   ```
   
   Additionally, if https://github.com/apache/airflow/issues/29746 is accepted it should accept `XComArg` and `XComPlainArg` too
   
   
   
   ### How to reproduce
   
   Mypy checks over this:
   
   ```python3
   from airflow import DAG
   from airflow.providers.databricks.operators.databricks import DatabricksSubmitRunOperator
   
   with DAG(
       "dag_erroring",
   ) as dag:
   
       DatabricksSubmitRunOperator(
           task_id="my_notebook_task",
           new_cluster={
               "cluster_name": "single-node-cluster",
               "spark_version": "7.6.x-scala2.12",
               "node_type_id": "i3.xlarge",
               "num_workers": 0,
               "spark_conf": {
                   "spark.databricks.cluster.profile": "singleNode",
                   "spark.master": "[*, 4]",
               },
               "custom_tags": {"ResourceClass": "SingleNode"},
           },
           notebook_task={
               "notebook_path": "some/path/to/a/notebook",
               "base_parameters": {"some_param": "abc1", "some_other_param": "abc2"},
           },
           libraries=[],
           databricks_retry_limit=3,
           timeout_seconds=86400,
           polling_period_seconds=20,
       )
   ```
   
   returns 
   `dags/my_dag_to_test.py:24: error: Dict entry 1 has incompatible type "str": "Dict[str, str]"; expected "str": "str"  [dict-item]`
   
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] alvaroqueiroz commented on issue #29931: Type hinting in `DatabricksSubmitRunOperator` is not compatible with parameter passing possibilities

["alvaroqueiroz (via GitHub)" <gi...@apache.org>]

alvaroqueiroz commented on issue #29931:
URL: https://github.com/apache/airflow/issues/29931#issuecomment-1655235595

   Is this still a problem?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #29931: Type hinting in `DatabricksSubmitRunOperator` is not compatible with parameter passing possibilities

["eladkal (via GitHub)" <gi...@apache.org>]

eladkal commented on issue #29931:
URL: https://github.com/apache/airflow/issues/29931#issuecomment-1464885962

   Feel free to raise PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org