You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Ravali-43 (via GitHub)" <gi...@apache.org> on 2024/05/21 11:01:36 UTC
[I] GCStoS3 Operator [airflow]
Ravali-43 opened a new issue, #39737:
URL: https://github.com/apache/airflow/issues/39737
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
_No response_
### Apache Airflow version
2.6.3
### Operating System
red hat linux 8.9
### Deployment
Google Cloud Composer
### Deployment details
_No response_
### What happened
trouble using GCStoS3 operator. while trying to copy the files from gs location to S3 bucket, instead of copying *.txt file from a folder, the current code is appending gcs file path to s3 location. Meaning resulting in having the entire file path in s3 destination as well. ex: parentfolder/childfolder/*.txt --> the same is copied into S3 as well.. s3://bucket/parentfolder/childfolder/*.txt instead s3://bucket/*.txt.
PS: in dag config i have set the keep_directory_structure to false. observed the same issue
### What you think should happen instead
_No response_
### How to reproduce
Please change the congif/code where only files are being copied to destination location
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GCStoS3 Operator [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #39737:
URL: https://github.com/apache/airflow/issues/39737#issuecomment-2155715349
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GCStoS3 Operator [airflow]
Posted by "hardikdedhia1992 (via GitHub)" <gi...@apache.org>.
hardikdedhia1992 commented on issue #39737:
URL: https://github.com/apache/airflow/issues/39737#issuecomment-2130142543
you can try with bashoperator use command such as gsutil -m rsync -r gs://gcs_bucket_path s3://sc_bucket_path with providing access to AWS in GCP . you can use :
rsync_task = BashOperator(task_id='rsync_task',
bash_command='command',
dag=dag)
Hope fully it should work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GCStoS3 Operator [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #39737:
URL: https://github.com/apache/airflow/issues/39737#issuecomment-2170984146
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GCStoS3 Operator [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #39737:
URL: https://github.com/apache/airflow/issues/39737#issuecomment-2122425243
Is this about Airflow?
I think you are expecting folder and file name structure but GCS doesn't really have folders. The "folder" is part of the object name.
> For example, if you create an object named folder1/file.txt in the bucket your-bucket, the path to the object is your-bucket/folder1/file.txt, and Cloud Storage has no folder named folder1 stored within it. From the perspective of Cloud Storage, the string folder1/ is part of the object's name.
https://cloud.google.com/storage/docs/objects
Thus from Airflow perspective `parentfolder/childfolder/.txt` is the file name.
If you think this behavior is flawed then this is a feature request to GCS not to Airflow.
You can always create custom operator to do the parsing you are looking for as this is something custom to your needs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GCStoS3 Operator [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #39737:
URL: https://github.com/apache/airflow/issues/39737#issuecomment-2122373794
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GCStoS3 Operator [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #39737: GCStoS3 Operator
URL: https://github.com/apache/airflow/issues/39737
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org