You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sdap.apache.org by GitBox <gi...@apache.org> on 2021/09/28 13:39:44 UTC
[GitHub] [incubator-sdap-ingester] wphyojpl opened a new pull request #43: wip: SDAP-TODO: feat: S3 obj store
wphyojpl opened a new pull request #43:
URL: https://github.com/apache/incubator-sdap-ingester/pull/43
- add class to store tile blob in S3 in addition to Cassandra
- update configuration to accept either S3 or Cassandra
- refactor main method to avoid code duplication
- TODO: add test case
- TODO: AWS credentials
### How SDAP handles S3 credentials
-
- Query repo already has S3 class to extract data from S3: https://github.com/apache/incubator-sdap-nexus/blob/master/data-access/nexustiles/dao/S3Proxy.py
- Collection Manager also has a class to observe files in S3 bucket to be ingested: https://github.com/apache/incubator-sdap-ingester/blob/dev/collection_manager/collection_manager/services/S3Observer.py
- Both classes assume AWS credentials are handled at a different place since they don't have code to do so.
- How are they being handled as we will need that info to write a test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-ingester] frankinspace commented on pull request #43: wip: SDAP-TODO: feat: S3 obj store
Posted by GitBox <gi...@apache.org>.
frankinspace commented on pull request #43:
URL: https://github.com/apache/incubator-sdap-ingester/pull/43#issuecomment-929364695
This was the "old" ingest code that was able to write to S3 instead of cassandra: https://github.com/apache/incubator-sdap-ningester/blob/master/src/main/java/org/apache/sdap/ningester/writer/S3Store.java
The analysis side as you see is able to read data from S3 instead of cassandra. I think @jjacob7734 might have worked on an instance of SDAP where that was configured.
As far as credentials go, it all uses the `boto3` library so you can provide credentials in a number of different ways:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
I don't recall off the top of my head but typically they would be provided either through environment variables or a ~/.aws/credentials file on the node running the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-ingester] jjacob7734 commented on pull request #43: wip: SDAP-TODO: feat: S3 obj store
Posted by GitBox <gi...@apache.org>.
jjacob7734 commented on pull request #43:
URL: https://github.com/apache/incubator-sdap-ingester/pull/43#issuecomment-929532976
Adding S3 as a datastore option was work done by David Silvan, a summer intern at JPL, in Summer of 2017. I think he worked primarily with @ngachung on that. @ngachung , Thomas Huang, @fgreg and I received an email from David on 8/9/2017 in which he attached some benchmarks comparing the S3 datastore to Cassandra. It looks like at that time his S3 implementation was 2x to 5x (maybe 3x on average) slower than Cassandra.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-ingester] jjacob7734 commented on pull request #43: wip: SDAP-TODO: feat: S3 obj store
Posted by GitBox <gi...@apache.org>.
jjacob7734 commented on pull request #43:
URL: https://github.com/apache/incubator-sdap-ingester/pull/43#issuecomment-929532976
Adding S3 as a datastore option was work done by David Silvan, a summer intern at JPL, in Summer of 2017. I think he worked primarily with @ngachung on that. @ngachung , Thomas Huang, @fgreg and I received an email from David on 8/9/2017 in which he attached some benchmarks comparing the S3 datastore to Cassandra. It looks like at that time his S3 implementation was 2x to 5x (maybe 3x on average) slower than Cassandra.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-ingester] frankinspace commented on pull request #43: wip: SDAP-TODO: feat: S3 obj store
Posted by GitBox <gi...@apache.org>.
frankinspace commented on pull request #43:
URL: https://github.com/apache/incubator-sdap-ingester/pull/43#issuecomment-929364695
This was the "old" ingest code that was able to write to S3 instead of cassandra: https://github.com/apache/incubator-sdap-ningester/blob/master/src/main/java/org/apache/sdap/ningester/writer/S3Store.java
The analysis side as you see is able to read data from S3 instead of cassandra. I think @jjacob7734 might have worked on an instance of SDAP where that was configured.
As far as credentials go, it all uses the `boto3` library so you can provide credentials in a number of different ways:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
I don't recall off the top of my head but typically they would be provided either through environment variables or a ~/.aws/credentials file on the node running the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org