You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "omrdyngc (via GitHub)" <gi...@apache.org> on 2025/06/16 11:45:29 UTC

[I] Microsoft AzureDataLakeStorageV2Hook has a kwargs handling bug [airflow]

omrdyngc opened a new issue, #51781:
URL: https://github.com/apache/airflow/issues/51781

   ### Apache Airflow Provider(s)
   
   microsoft-azure
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-microsoft-azure==12.3.1
   
   ### Apache Airflow version
   
   apache/airflow:3.0.1-python3.12
   
   ### Operating System
   
   PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
   
   ### Deployment
   
   Other Docker-based deployment
   
   ### Deployment details
   
   I used this Docker file to deploy Airflow to Azure Container Instance
   
   FROM apache/airflow:3.0.1-python3.12
   USER root
   RUN apt-get update && \
       apt-get install -y --no-install-recommends \
           build-essential \
           git \
       && apt-get autoremove -yqq --purge \
       && apt-get clean \
       && rm -rf /var/lib/apt/lists/*
   USER airflow
   ENV AIRFLOW_VERSION=3.0.1
   ENV PYTHON_VERSION=3.12
   RUN curl -sSL "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt" \
       -o /tmp/constraints.txt
   RUN pip install --no-cache-dir \
       apache-airflow-providers-microsoft-azure \
       apache-airflow-providers-common-sql \
       apache-airflow-providers-postgres \
       apache-airflow-providers-git \
       psycopg2-binary \
       azure-identity \
       azure-storage-file-datalake \
       --constraint /tmp/constraints.txt
   
   ### What happened
   
   def upload_file(self,file_system_name: FileSystemProperties | str, file_name: str,file_path: str, overwrite: bool = False,
           **kwargs: Any,) -> None:
   
           file_client = self.create_file(file_system_name, file_name)
           with open(file_path, "rb") as data:
               file_client.upload_data(data, overwrite=overwrite, kwargs=kwargs)
   
   The function from AzureDataLakeStorageV2Hook class is passing kwargs wrong and causing the below error when trying to upload
   
   
    ERROR - Task failed with exception: source="task"
   TypeError: Session.request() got an unexpected keyword argument 'kwargs'
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 838 in run
   
   ### What you think should happen instead
   
   file_client.upload_data(data, overwrite=overwrite, kwargs=kwargs) should be  
   file_client.upload_data(data, overwrite=overwrite, **kwargs)  
   
   Because the function is not unpacking the dictionary; instead, it is passing kwargs as a single argument called kwargs.
   
   ### How to reproduce
   
     
     @task
       def upload_to_adls_not_working():
           hook = AzureDataLakeStorageV2Hook(adls_conn_id='DLAKEGEN2')
           file_content = json.dumps({"message": "Hello, World!"})
   
           with tempfile.NamedTemporaryFile(delete=False, suffix=".json") as tmp:
             tmp.write(file_content.encode("utf-8"))
             tmp_path = tmp.name
   
           hook.upload_file(
            file_system_name='xxxxxxx',
            file_name="upload_to_adls_not_working.json",
            file_path=tmp_path,
            overwrite=True,
           )
   
   ### Anything else
   
   same bug also in upload_file_to_directory() and create_directory()
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Microsoft AzureDataLakeStorageV2Hook has a kwargs handling bug [airflow]

Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #51781:
URL: https://github.com/apache/airflow/issues/51781#issuecomment-2976700970

   I see that you're open to raising a PR for this issue - I went ahead and assigned it to you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Microsoft AzureDataLakeStorageV2Hook has a kwargs handling bug [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W closed issue #51781: Microsoft AzureDataLakeStorageV2Hook has a kwargs handling bug
URL: https://github.com/apache/airflow/issues/51781


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Microsoft AzureDataLakeStorageV2Hook has a kwargs handling bug [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #51781:
URL: https://github.com/apache/airflow/issues/51781#issuecomment-2976232192

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org