[Announce] S3gateway-gRPC HDDS-4440 Proof of Concept (+ branch creation)

[Neil Joshi <ne...@gmail.com>]

Hi,

We recently created a proof-of-concept, POC, implementation for processing
S3 gateway, s3g, requests with the transport of OmRequests over gRPC.  This
follows HDDS-4440
https://ozone.apache.org/docs/1.1.0/design/s3-performance.html.

Current commit implementation can be found in:
https://github.com/neils-dev/ozone/tree/HDDS-4440-s3-performance.  Slack
channel #ozone-s3g-grpc .

The implementation processes S3 gateway requests through a gRPC
OmTransport, a design that allows a constant OzoneClient connection and aws
signature authentication on a per request basis.  This differs from the
current implementation relying on HadoopRPC that creates an OzoneClient
connection for each request.  The need to create a 'client' connection for
each request results in overhead impacting request processing times.
Another limitation to the existing HadoopRPC based OmTransport
implementation is that errors in request processing are shown as ambiguous
Http 500 server responses to S3 gateway callers.  With the redesigned s3g,
proper error responses describing the error can be relayed to the caller.

Preliminary performance comparisons between the existing s3g HadoopRPC
OmTransport and the POC gRPC OmTransport show at least a 2x improvement in
request processing rates for simple file operations (create buckets).  See
slack channel #ozone-s3g-gprc for more info and inquiries.

Branch HDDS-4440-s3-performance is also created.

Please let us know, if you have any comments/questions...

Best -
Neil & Jasen

G-Research
-- 
NJ