Introduce a new SPI for storing Attachments aka AttachmentStore

[Chetan Mehrotra <ch...@gmail.com>]

Currently the code data for actions are stored as attachments in
CouchDB. For larger binaries (say > 1MB) it would be better to store
them in some binary store. Further if we are deploying a custom OW
setup in cloud then it would be better to store the binaries in cloud
specific binary storage like S3/Azure BlobStore/IBM Object Storage

To support such a case we should introduce a new SPI in OW
`AttachmentStore`. Some key points wrt to such an spi

1. Used internally by ArtifactStore and need not be exposed to layers above.

2. A default memory based implementation along with TCK to validate the contract

3. Later implement custom implementation for S3/Azure BlobStore/IBM
Object Storage which can use the newly refactored PoolingRestClient
and have the TCK test run against them

4. Allow using such a stored for CouchDbRestStore also via config

The interface can possibly look like [1]

Thoughts?

Chetan Mehrotra
[1] https://github.com/chetanmeh/openwhisk-mongo/blob/master/common/mongo/src/main/scala/whisk/core/database/mongo/AttachmentStore.scala

Re: Introduce a new SPI for storing Attachments aka AttachmentStore

[Rodric Rabbah <ro...@gmail.com>]

Thanks for plowing through on this - I reviewed the initial PR and the
interface looks good.
I am still grokking the implementation though, I pinged you on slack.

-r

On Wed, Feb 28, 2018 at 9:06 AM, Chetan Mehrotra <ch...@gmail.com>
wrote:

> Currently the code data for actions are stored as attachments in
> CouchDB. For larger binaries (say > 1MB) it would be better to store
> them in some binary store. Further if we are deploying a custom OW
> setup in cloud then it would be better to store the binaries in cloud
> specific binary storage like S3/Azure BlobStore/IBM Object Storage
>
> To support such a case we should introduce a new SPI in OW
> `AttachmentStore`. Some key points wrt to such an spi
>
> 1. Used internally by ArtifactStore and need not be exposed to layers
> above.
>
> 2. A default memory based implementation along with TCK to validate the
> contract
>
> 3. Later implement custom implementation for S3/Azure BlobStore/IBM
> Object Storage which can use the newly refactored PoolingRestClient
> and have the TCK test run against them
>
> 4. Allow using such a stored for CouchDbRestStore also via config
>
> The interface can possibly look like [1]
>
> Thoughts?
>
> Chetan Mehrotra
> [1] https://github.com/chetanmeh/openwhisk-mongo/blob/master/
> common/mongo/src/main/scala/whisk/core/database/mongo/
> AttachmentStore.scala
>