[jira] [Work logged] (BEAM-13399) [Go SDK] Add automated expansion service start up feature

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-13399?focusedWorklogId=718351&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-718351 ]

ASF GitHub Bot logged work on BEAM-13399:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 01/Feb/22 00:35
            Start Date: 01/Feb/22 00:35
    Worklog Time Spent: 10m 
      Work Description: lostluck commented on a change in pull request #16671:
URL: https://github.com/apache/beam/pull/16671#discussion_r796167396



##########
File path: sdks/go/pkg/beam/transforms/sql/sql.go
##########
@@ -79,6 +82,33 @@ func ExpansionAddr(addr string) Option {
 	}
 }
 
+const serviceGradleTarget = ":sdks:java:extensions:sql:expansion-service:shadowJar"
+
+// StartAutomatedExpansionService pulls a JAR from Maven containing
+// a Beam expansion service that handles the SQL transform URN,
+// creates a runner type to manage the service, starts the service,
+// and arranges the endpoint with the appropriate namespace tag.
+//
+// Returns the StopService() function for the runner (should be
+// deferred so the service is stopped before the program exits,)
+// the tagged expansion address, and an error.
+func StartAutomatedExpansionService() (func() error, string, error) {

Review comment:
       I think my question is why is this all done at the wrapper package level, rather than hidden away deeper? As implemented, every Jar based Xlang wrapper has to duplicate all this code and the code at the transform call itself, which is not a great experience, especially since there's nothing specific about it.
   
   Ideally, wrapper authors should only be writing
   
   defaultExpansion := xlangx.AutoNamespace + xlangx.Separator + <gradle target>
   
   Or similar (or a wrapper function like Require does).
   
   Then they pass that in, and the auto startup code will take care of the rest. Kinda like how the Python versions just take in the gradle target, and then the code takes care of the rest. In this case, everything should be happening within the scope of the auto HandlerFunc.
   
   We do not need to optimize for repeated calls to the same expansion service. Users with complex enough pipelines hitting the same service should likely be standing up their own anyway.
   
   If we don't want to use up "auto" just for jar startup we can always have something else to specify what kind of auto thing it's starting up: eg. "auto:jar:" to have it select sub name space for handling, or "autojar:" and just lean on the existing selector.

##########
File path: sdks/go/pkg/beam/core/runtime/xlangx/expand.go
##########
@@ -140,3 +140,45 @@ func QueryExpansionService(ctx context.Context, p *HandlerParams) (*jobpb.Expans
 	}
 	return res, nil
 }
+
+// QueryAutomatedExpansionService submits an external transform to be expanded by the
+// expansion service and then eagerly materializes the artifacts for staging. The given
+// transform should be the external transform, and the componenets are any additional

Review comment:
       ```suggestion
   // transform should be the external transform, and the components are any additional
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 718351)
    Time Spent: 17h 40m  (was: 17.5h)

> [Go SDK] Add automated expansion service start up feature
> ---------------------------------------------------------
>
>                 Key: BEAM-13399
>                 URL: https://issues.apache.org/jira/browse/BEAM-13399
>             Project: Beam
>          Issue Type: Improvement
>          Components: cross-language, sdk-go
>            Reporter: Jack McCluskey
>            Assignee: Jack McCluskey
>            Priority: P2
>          Time Spent: 17h 40m
>  Remaining Estimate: 0h
>
> Add Go SDK support for automated expansion service start-up if one is not provided, following the example of the Python implementation.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)