[GitHub] [pulsar] jerrypeng opened a new issue #4619: Improve REST API for Pulsar Functions/Sources/Sinks

[GitBox <gi...@apache.org>]

jerrypeng opened a new issue #4619: Improve REST API for Pulsar Functions/Sources/Sinks
URL: https://github.com/apache/pulsar/issues/4619
 
 
   Currently, the REST API for creating and updating Pulsar Functions/Sources/Sinks is complicated and hard for users call.  There are many awkward aspects of the current REST API that complicates using the REST API
   
   1. The current REST API requires parameters to submitted via FORM data which is non-standard for REST Interfaces since the FORM data is usually used when submitting data from forms on a website.
   
   2. Parameters, like packageUrl and FunctionConfig/SourceConfig/SinkConfig, are submitted as separate Form Data which is also confusing and non-standard
   
   
   Currently to submit a built-in source via REST:
   
   ```
   curl  --request POST  -H "Content-Type: multipart/form-data" -F sourceConfig='{"tenant":"jerry","namespace":"default","name":"test-source","className":"","topicName":"jerry/default/source","parallelism":1,"archive":"builtin://data-generator"};type=application/json' localhost:8080/admin/v3/source/public/default/test-source
   ```
   
   These details of the curl can be very hard for a user to get right
   
   Ideally we have a REST interface that just takes a single JSON payload e.g.
   
   ```
   curl \
       --request POST \
       --data @payload.json \
        localhost:8080/admin/v3/source/public/default/test-source
   ```
   
   The REST API will be much easier to use and document if we collapse all the parameters that need to be submitted into a single JSON payload.  The function/source/sink binary package can even be included in the JSON
   
   AWS Lambda Functions and Google Cloud Functions have similar REST APIs to what I am proposing
   
   https://docs.aws.amazon.com/lambda/latest/dg/API_CreateFunction.html
   
   https://cloud.google.com/functions/docs/reference/rest/v1/projects.locations.functions#CloudFunction
   
   A user should be submit a JSON payload something like the following to create or update a function/source or sink.
   
   ``` 
   {
   	"package": {
   		"url": "<URL>",
   		"binary": "...."
   	},
   	"name": "jerry-function",
   	"tenant": "public",
   	"namespace": "default",
   	"inputs": [
   		{
   			"topic": "persistent://public/default/input-1",
   			"receiverQueueSize": 1000
   		},
   		{
   			"topic": "persistent://public/default/input-2",
   			"receiverQueueSize": 500
   		}
   	],
   	"output": "persistent://public/default/jerry-output",
   	"parallelism": 2
   }
   ```
   
   There might be some complications if we want to be able to submit JSON payload to also contain the function package as the package can be large. We will have to stream the JSON to the server and parse the json in a stream like fashion on the server side as well to keep memory usage in check.  I think Jackson already supports such as use pattern:
   
   https://www.baeldung.com/jackson-streaming-api

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services