You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "1inuxoid (via GitHub)" <gi...@apache.org> on 2023/02/08 08:58:28 UTC

[GitHub] [airflow] 1inuxoid opened a new issue, #29422: Multiple AWS connections support in DynamoDBToS3Operator

1inuxoid opened a new issue, #29422:
URL: https://github.com/apache/airflow/issues/29422

   ### Description
   
   I want to add support of a separate AWS connection for DynamoDB in `DynamoDBToS3Operator` via `aws_dynamodb_conn_id` constructor argument.
   
   
   
   ### Use case/motivation
   
   Sometimes DynamoDB tables and S3 buckets live in different AWS accounts so to access both resources you need to assume a role in another account from one of them.
   That role can be specified in AWS connection, thus we need to support two of them in this operator.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] 1inuxoid commented on issue #29422: Multiple AWS connections support in DynamoDBToS3Operator

Posted by "1inuxoid (via GitHub)" <gi...@apache.org>.
1inuxoid commented on issue #29422:
URL: https://github.com/apache/airflow/issues/29422#issuecomment-1424815215

   @eladkal, I can see that `RedshiftToS3Operator` and `S3ToRedshiftOperator` already implement a similar pattern using `redshift_conn_id` connection argument.
   I don't see any other operators in this "transfer family".
   
   Also, I think the semantics of two AWS connections in case of such `AwsToAwsBaseTransferOperator` would be tricky and not straightforward at all. How would you even name them if they are not service-specific? 
   
   I'll probably continue with a change to `DynamoDBToS3Operator` specifically for now, however, I'm open to suggestions on how to address this semantical challenge. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] o-nikolas commented on issue #29422: Multiple AWS connections support in DynamoDBToS3Operator

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on issue #29422:
URL: https://github.com/apache/airflow/issues/29422#issuecomment-1424875593

   @eladkal Are you thinking of something like a generic "source" and "destination" connection that would be implemented in a base AWS transfer operator that all concrete transfer operators would use? Would both those connections be based on the aws connection? I suppose we could use extras to plumb anything service specific through. It's an interesting thought.
   
   What do you think @Taragolis?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] Taragolis closed issue #29422: Multiple AWS connections support in DynamoDBToS3Operator

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis closed issue #29422: Multiple AWS connections support in DynamoDBToS3Operator
URL: https://github.com/apache/airflow/issues/29422


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #29422: Multiple AWS connections support in DynamoDBToS3Operator

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #29422:
URL: https://github.com/apache/airflow/issues/29422#issuecomment-1422797947

   I think this is relevant for any aws to aws transfer operator (and we have several of thouse)
   I wonder if we can have a generic solution.
   cc @o-nikolas @shubham22


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29422: Multiple AWS connections support in DynamoDBToS3Operator

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29422:
URL: https://github.com/apache/airflow/issues/29422#issuecomment-1422251357

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org