You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devlake.apache.org by "artofcomputing (via GitHub)" <gi...@apache.org> on 2023/08/21 18:53:33 UTC
[GitHub] [incubator-devlake] artofcomputing opened a new issue, #5930: [Feature][Object Storage] Allow Domain Data to be written in object storage
artofcomputing opened a new issue, #5930:
URL: https://github.com/apache/incubator-devlake/issues/5930
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar feature requirement.
### Use case
As a Data Engineer, I want the flexibility to:
- Save Domain-Data in Object Storage to be consumed by other systems;
- Choose the data format to be written in Object Storage (e.g CSV, PARQUET, TSV);
- Simplify DB management (by only keeping system tables in a OLTP system);
Currently, Apache DevLake doesn't have these features natively.
### Description
By having these, any team can easily integrate Apache DevLake in their current ETL stack, and it will let teams choose the best way to persist Domain Data within their systems.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Object Storage] Allow Domain Data to be written in object storage [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5930:
URL: https://github.com/apache/incubator-devlake/issues/5930#issuecomment-1784288213
This issue has been closed because it has been inactive for a long time. You can reopen it if you encounter the similar problem in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #5930: [Feature][Object Storage] Allow Domain Data to be written in object storage
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on issue #5930:
URL: https://github.com/apache/incubator-devlake/issues/5930#issuecomment-1687322962
Hi @artofcomputing , I'm open to this. Can you tell me a specific use case of how you'll use DevLake with another data format?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] ZhangNing10 commented on issue #5930: [Feature][Object Storage] Allow Domain Data to be written in object storage
Posted by "ZhangNing10 (via GitHub)" <gi...@apache.org>.
ZhangNing10 commented on issue #5930:
URL: https://github.com/apache/incubator-devlake/issues/5930#issuecomment-1687298957
@artofcomputing thanks for reporting! @klesh @Startrekzky could you kindly have a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Object Storage] Allow Domain Data to be written in object storage [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5930:
URL: https://github.com/apache/incubator-devlake/issues/5930#issuecomment-1774244276
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] artofcomputing commented on issue #5930: [Feature][Object Storage] Allow Domain Data to be written in object storage
Posted by "artofcomputing (via GitHub)" <gi...@apache.org>.
artofcomputing commented on issue #5930:
URL: https://github.com/apache/incubator-devlake/issues/5930#issuecomment-1688170553
Thanks @Startrekzky!
My current use-case would be:
- Output Domain Layer Data to CSV/Parquet in a S3 bucket
- Consume S3 CSV/Parquet files with a Apache Airflow DAG
- Load data into Google Big Query within the same Airflow DAG
However, by allowing Domain Layer Data to be exported to object storage (AWS S3, GCS, ABS) the possibilities are endless (I can choose X number of ETL tools to extract this data and load it into Y numbers of OLAP systems).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Object Storage] Allow Domain Data to be written in object storage [incubator-devlake]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #5930: [Feature][Object Storage] Allow Domain Data to be written in object storage
URL: https://github.com/apache/incubator-devlake/issues/5930
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #5930: [Feature][Object Storage] Allow Domain Data to be written in object storage
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on issue #5930:
URL: https://github.com/apache/incubator-devlake/issues/5930#issuecomment-1689631999
Hi @artofcomputing. Thank you for the feedback. I'll consider it. Just to clarify that it might not be supported in the short term as it requires a lot of work. The community has higher priorities which you can find in the [roadmap](https://devlake.apache.org/docs/Overview/Roadmap)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] artofcomputing commented on issue #5930: [Feature][Object Storage] Allow Domain Data to be written in object storage
Posted by "artofcomputing (via GitHub)" <gi...@apache.org>.
artofcomputing commented on issue #5930:
URL: https://github.com/apache/incubator-devlake/issues/5930#issuecomment-1689923913
No worries @Startrekzky, it'll be a great addition to DevLake in a long term, for now I'll extract the data from the OLTP DB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org