[GitHub] [airflow] ecerulm opened a new issue #15761: Raise KubernetesPodOperator do_xcom_push size limit

[GitBox <gi...@apache.org>]

ecerulm opened a new issue #15761:
URL: https://github.com/apache/airflow/issues/15761


   <!--
   
   Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
   Don't worry if they're not all applicable; just try to include what you can :-)
   
   If you need to include code snippets or logs, please put them in fenced code
   blocks.  If they're super-long, please use the details tag like
   <details><summary>super-long log</summary> lots of stuff </details>
   
   Please delete these comment blocks before submitting the issue.
   
   -->
   
   **Description**
   
   Currently KubernetesPodOperator will push the file `/airflow/xcom/return.json` as  XCom value.
   This value is affected by two limits
   
   * The size limit for the xcom value set by the airflow default built-in xcom backend, but you can provide *a custom XCom backend that does not have that limit*, like an backend that uploads values to S3
   * The amount of bytes that can be transferred in a single request from kubernetes stream. (or at least that's what I suspect)
   
   I would expect to be able to get the full contents of the file at the custom Xcom backend
   
   
   **Use case / motivation**
   
   I want to receive the full contents of `/airflow/xcom/return.json` as XCom value into my custom Xcom backend (that will upload to S3). 
   
   
   Otherwise I need to provide a custom logic in my docker image to upload to S3 instead with the following drawback
   * no secure way to pass the aws connection credentials from airflow to the kubernetespodoperator
   * duplication of code. 
   
   
   
   
   
   **Are you willing to submit a PR?**
   
   <!--- We accept contributions! -->
   
   **Related Issues**
   
   <!-- Is there currently another issue associated with this? -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #15761: Raise KubernetesPodOperator do_xcom_push size limit

[GitBox <gi...@apache.org>]

eladkal commented on issue #15761:
URL: https://github.com/apache/airflow/issues/15761#issuecomment-864521308


   > no secure way to pass the aws connection credentials from airflow to the kubernetespodoperator
   
   This has some similarities with the idea in https://github.com/apache/airflow/issues/16237


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org