You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/04/07 08:37:50 UTC

[GitHub] [pulsar] jiazhai opened a new pull request #6684: Support function with format: Function>

jiazhai opened a new pull request #6684: Support function with format: Function<I, CompletableFuture<O>>
URL: https://github.com/apache/pulsar/pull/6684
 
 
   Fixes #6519
   
   ### Motivation
   
   Currently, Pulsar Functions not support Async mode, and if user passed in a Function in format :
   ```
   Function<I, CompletableFuture<O>>
   ```
   This kind of function is useful if the function might use RPCs to call external systems.
   
   e.g.
   ```java
   public class AsyncFunction implements Function<String, CompletableFuture<O>> {
       CompletableFuture<O> apply (String input) {
           CompletableFuture future = new CompletableFuture();
           ...function compute...
           future.whenComplete(() -> {
               ...  call external system  ...
           });
           return future;
       }
   ```
   
   ### Modifications
   - add support for Async Functions support.
   
   ### Verifying this change
   current ut passed.
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] sijie commented on a change in pull request #6684: Support function with format: Function>

Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #6684: Support function with format: Function<I, CompletableFuture<O>>
URL: https://github.com/apache/pulsar/pull/6684#discussion_r404987985
 
 

 ##########
 File path: pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstance.java
 ##########
 @@ -52,23 +53,49 @@ public JavaInstance(ContextImpl contextImpl, Object userClassObject) {
         }
     }
 
-    public JavaExecutionResult handleMessage(Record<?> record, Object input) {
+    public CompletableFuture<JavaExecutionResult> handleMessage(Record<?> record, Object input) {
         if (context != null) {
             context.setCurrentMessageContext(record);
         }
+
+        final CompletableFuture<JavaExecutionResult> future = new CompletableFuture<>();
         JavaExecutionResult executionResult = new JavaExecutionResult();
+
         try {
             Object output;
             if (function != null) {
                 output = function.process(input, context);
             } else {
                 output = javaUtilFunction.apply(input);
             }
-            executionResult.setResult(output);
+
+            if (output instanceof CompletableFuture) {
+                // Function is in format: Function<I, CompletableFuture<O>>
+                ((CompletableFuture) output).whenComplete((obj, throwable) -> {
 
 Review comment:
   shared the same concern here. I think we need to introduce a setting to limit the max outstanding items (results) being processed. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] merlimat commented on a change in pull request #6684: Support function with format: Function>

Posted by GitBox <gi...@apache.org>.
merlimat commented on a change in pull request #6684: Support function with format: Function<I, CompletableFuture<O>>
URL: https://github.com/apache/pulsar/pull/6684#discussion_r404978597
 
 

 ##########
 File path: pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstance.java
 ##########
 @@ -52,23 +53,49 @@ public JavaInstance(ContextImpl contextImpl, Object userClassObject) {
         }
     }
 
-    public JavaExecutionResult handleMessage(Record<?> record, Object input) {
+    public CompletableFuture<JavaExecutionResult> handleMessage(Record<?> record, Object input) {
         if (context != null) {
             context.setCurrentMessageContext(record);
         }
+
+        final CompletableFuture<JavaExecutionResult> future = new CompletableFuture<>();
         JavaExecutionResult executionResult = new JavaExecutionResult();
+
         try {
             Object output;
             if (function != null) {
                 output = function.process(input, context);
             } else {
                 output = javaUtilFunction.apply(input);
             }
-            executionResult.setResult(output);
+
+            if (output instanceof CompletableFuture) {
+                // Function is in format: Function<I, CompletableFuture<O>>
+                ((CompletableFuture) output).whenComplete((obj, throwable) -> {
 
 Review comment:
   This is not applying any backpressure on the consumer. If the function doesn't have a mechanism to self-throttle, it will just blow up the memory.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] jiazhai commented on a change in pull request #6684: Support function with format: Function>

Posted by GitBox <gi...@apache.org>.
jiazhai commented on a change in pull request #6684: Support function with format: Function<I, CompletableFuture<O>>
URL: https://github.com/apache/pulsar/pull/6684#discussion_r405216485
 
 

 ##########
 File path: pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstance.java
 ##########
 @@ -52,23 +53,49 @@ public JavaInstance(ContextImpl contextImpl, Object userClassObject) {
         }
     }
 
-    public JavaExecutionResult handleMessage(Record<?> record, Object input) {
+    public CompletableFuture<JavaExecutionResult> handleMessage(Record<?> record, Object input) {
         if (context != null) {
             context.setCurrentMessageContext(record);
         }
+
+        final CompletableFuture<JavaExecutionResult> future = new CompletableFuture<>();
         JavaExecutionResult executionResult = new JavaExecutionResult();
+
         try {
             Object output;
             if (function != null) {
                 output = function.process(input, context);
             } else {
                 output = javaUtilFunction.apply(input);
             }
-            executionResult.setResult(output);
+
+            if (output instanceof CompletableFuture) {
+                // Function is in format: Function<I, CompletableFuture<O>>
+                ((CompletableFuture) output).whenComplete((obj, throwable) -> {
 
 Review comment:
   Ok. Thanks for the suggestion.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services