You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/10/07 21:33:25 UTC
[GitHub] [pulsar] flowchartsman opened a new issue #8216: Go SDK: GetUserConfMap() is not getting populated
flowchartsman opened a new issue #8216:
URL: https://github.com/apache/pulsar/issues/8216
**Describe the bug**
While the user config is listed as not being implemented for the Pulsar Functions SDK for go, the examples seem to indicate that I should be able to poll the KVs through the context function, specifically:
```
func (c *FunctionContext) GetUserConfValue(key string) interface{} {
return c.userConfigs[key]
}
func (c *FunctionContext) GetUserConfMap() map[string]interface{} {
return c.userConfigs
}
```
However, these do not appear to be getting populated.
**To Reproduce**
Deploy a pulsar function for go with the following code in main.go:
```go
package main
import (
"context"
log "github.com/apache/pulsar/pulsar-function-go/logutil"
"github.com/apache/pulsar/pulsar-function-go/pf"
)
func contextFunc(ctx context.Context) {
if fc, ok := pf.FromContext(ctx); ok {
log.Infof("function ID is:%s, ", fc.GetFuncID())
log.Infof("function version is:%s\n", fc.GetFuncVersion())
for k := range fc.GetUserConfMap() {
log.Infof("Config %q -> \"%v\"", k, fc.GetUserConfValue(k))
}
}
}
func main() {
pf.Start(contextFunc)
}
```
```
$ pulsar-admin functions create --tenant public --namespace default --name pfunctestconfig --go (pwd)/pfunctest --inputs persistent://public/default/in --output persistent://public/default/out --user-config '{"test":"value","testnum":1}'
```
**Expected behavior**
Logs look like:
```
2020/10/07 21:27:26.022 log.go:46: [info] function ID is:3cbefae6-5a6a-43c5-8427-148625d5f391,
2020/10/07 21:27:26.022 log.go:46: [info] function version is:e215cf0e-2a64-498c-8773-760173de11c3
2020/10/07 21:27:26.022 log.go:46: [info] Config "test" -> "value"
2020/10/07 21:27:26.022 log.go:46: [info] Config "testnum" -> "1"
```
**Actual behavior**
```
2020/10/07 21:27:26.022 log.go:46: [info] function ID is:3cbefae6-5a6a-43c5-8427-148625d5f391,
2020/10/07 21:27:26.022 log.go:46: [info] function version is:e215cf0e-2a64-498c-8773-760173de11c3
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wolfstudy commented on issue #8216: Go SDK: GetUserConfMap() is not getting populated
Posted by GitBox <gi...@apache.org>.
wolfstudy commented on issue #8216:
URL: https://github.com/apache/pulsar/issues/8216#issuecomment-738700753
@flowchartsman It seems that #8780 is already solving this problem, can you help to check whether we can close this issue, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] vaihtovirta commented on issue #8216: Go SDK: GetUserConfMap() is not getting populated
Posted by GitBox <gi...@apache.org>.
vaihtovirta commented on issue #8216:
URL: https://github.com/apache/pulsar/issues/8216#issuecomment-721313544
Hey @flowchartsman
I added user config support in my PR some time ago https://github.com/apache/pulsar/pull/8290
Have you checked if this solves the issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] vaihtovirta commented on issue #8216: Go SDK: GetUserConfMap() is not getting populated
Posted by GitBox <gi...@apache.org>.
vaihtovirta commented on issue #8216:
URL: https://github.com/apache/pulsar/issues/8216#issuecomment-736772752
Hey @flowchartsman
It seems the change has been introduced here: [k8s runtime with go functions support #8352](https://github.com/apache/pulsar/pull/8352/files#diff-aafe8947b1fae9aec06d90c14db1b3b9fe720eb2a55ca48bfe7b43ecfb39efb4R231)
Also, it looks like python and java functions use a similar technique, but process configuration string through `JsonFormat.printer()`:
https://github.com/apache/pulsar/blob/a045d926c9620a3744a4bc755bf6f2a65209e6b6/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/RuntimeUtils.java#L349
I'll check if it's possible to handle golang runtime the same way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] flowchartsman edited a comment on issue #8216: Go SDK: GetUserConfMap() is not getting populated
Posted by GitBox <gi...@apache.org>.
flowchartsman edited a comment on issue #8216:
URL: https://github.com/apache/pulsar/issues/8216#issuecomment-736224054
@vaihtovirta I have discovered the issue. There are spurious single quotes (`'`) are getting added to the arguments to the pulsar function, causing it to fail parsing. The offending line is located in `pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/RuntimeUtils.java`:
```java
args.add("'" + configContent + "'");
```
Should be:
```java
args.add(configContent);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] flowchartsman edited a comment on issue #8216: Go SDK: GetUserConfMap() is not getting populated
Posted by GitBox <gi...@apache.org>.
flowchartsman edited a comment on issue #8216:
URL: https://github.com/apache/pulsar/issues/8216#issuecomment-736171055
Testing has been unsuccessful so far, but I may be doing something wrong. Currently testing #8365 with `--user-config '{"foo":"bar"}'` but getting
```
02:44:49.597 [function-timer-thread-106-1] INFO org.apache.pulsar.functions.runtime.process.ProcessRuntime - Started process successfully 2020/12/01 02:44:49.603 log.go:46: [info] The default config file path is: conf/conf.yaml
2020/12/01 02:44:49.605 log.go:76: [error] unmarshal config content error:invalid character '\'' looking for beginning of value panic: config file is nil. goroutine 1 [running]:
github.com/apache/pulsar/pulsar-function-go/pf.newInstanceConf(0x62d40) /<home>/go/pkg/mod/github.com/apache/pulsar/pulsar-function-go@v0.0.0-20201130103018-9c28378aea7d/pf/instanceConf.go:49 +0x646 github.com/apache/pulsar/pulsar-function-go/pf.NewFuncContext(0x0)
/<home>/go/pkg/mod/github.com/apache/pulsar/pulsar-function-go@v0.0.0-20201130103018-9c28378aea7d/pf/context.go:44 +0x26 github.com/apache/pulsar/pulsar-function-go/pf.newGoInstance(0xbbdd40) /<home>/go/pkg/mod/github.com/apache/pulsar/pulsar-function-go@v0.0.0-20201130103018-9c28378aea7d/pf/instance.go:65 +0x37
github.com/apache/pulsar/pulsar-function-go/pf.Start(0xbbdd40, 0xce1918) /<home>/go/pkg/mod/github.com/apache/pulsar/pulsar-function-go@v0.0.0-20201130103018-9c28378aea7d/pf/function.go:170 +0x52
main.main() /<home>/go/src/scratch/pulsartest/cmd/function/main.go:62 +0x39
```
where `main.go:62` corresponds to:
```go
pf.Start(contextFunc)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] flowchartsman edited a comment on issue #8216: Go SDK: GetUserConfMap() is not getting populated
Posted by GitBox <gi...@apache.org>.
flowchartsman edited a comment on issue #8216:
URL: https://github.com/apache/pulsar/issues/8216#issuecomment-736224054
@vaihtovirta I have discovered the issue. There are spurious single quotes (`'`) are getting added to the arguments to the pulsar function, causing it to fail parsing. The offending line is located in `pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/RuntimeUtils.java`:
```java
args.add("'" + configContent + "'");
```
Should be:
```java
args.add(configContent);
```
I will file an issue and mention you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] flowchartsman edited a comment on issue #8216: Go SDK: GetUserConfMap() is not getting populated
Posted by GitBox <gi...@apache.org>.
flowchartsman edited a comment on issue #8216:
URL: https://github.com/apache/pulsar/issues/8216#issuecomment-736171055
Testing has been unsuccessful so far, but I may be doing something wrong. Currently testing #8365 with `--user-config '{"foo":"bar"}'` but getting:
```
02:44:49.597 [function-timer-thread-106-1] INFO org.apache.pulsar.functions.runtime.process.ProcessRuntime - Started process successfully
2020/12/01 02:44:49.603 log.go:46: [info] The default config file path is: conf/conf.yaml
2020/12/01 02:44:49.605 log.go:76: [error] unmarshal config content error:invalid character '\'' looking for beginning of value
panic: config file is nil.
goroutine 1 [running]:
github.com/apache/pulsar/pulsar-function-go/pf.newInstanceConf(0x62d40)
/homedir/go/pkg/mod/github.com/apache/pulsar/pulsar-function-go@v0.0.0-20201130103018-9c28378aea7d/pf/instanceConf.go:49 +0x646
github.com/apache/pulsar/pulsar-function-go/pf.NewFuncContext(0x0)
/homedir/go/pkg/mod/github.com/apache/pulsar/pulsar-function-go@v0.0.0-20201130103018-9c28378aea7d/pf/context.go:44 +0x26
github.com/apache/pulsar/pulsar-function-go/pf.newGoInstance(0xbbdd40)
/homedir/go/pkg/mod/github.com/apache/pulsar/pulsar-function-go@v0.0.0-20201130103018-9c28378aea7d/pf/instance.go:65 +0x37
github.com/apache/pulsar/pulsar-function-go/pf.Start(0xbbdd40, 0xce1918)
/hoomedir/go/pkg/mod/github.com/apache/pulsar/pulsar-function-go@v0.0.0-20201130103018-9c28378aea7d/pf/function.go:170 +0x52
main.main()
/homedir/go/src/scratch/pulsartest/cmd/function/main.go:62 +0x39
```
where `main.go:62` corresponds to:
```go
pf.Start(contextFunc)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] flowchartsman commented on issue #8216: Go SDK: GetUserConfMap() is not getting populated
Posted by GitBox <gi...@apache.org>.
flowchartsman commented on issue #8216:
URL: https://github.com/apache/pulsar/issues/8216#issuecomment-736171055
Testing has been unsuccessful so far, but I may be doing something wrong. Currently testing #8365 with `--user-config '{"foo":"bar"}'` but getting
```
log.go:76:` [error] unmarshal config content error:invalid character '\'' looking for beginning of value
panic: config file is nil.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wolfstudy closed issue #8216: Go SDK: GetUserConfMap() is not getting populated
Posted by GitBox <gi...@apache.org>.
wolfstudy closed issue #8216:
URL: https://github.com/apache/pulsar/issues/8216
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] flowchartsman edited a comment on issue #8216: Go SDK: GetUserConfMap() is not getting populated
Posted by GitBox <gi...@apache.org>.
flowchartsman edited a comment on issue #8216:
URL: https://github.com/apache/pulsar/issues/8216#issuecomment-736171055
Testing has been unsuccessful so far, but I may be doing something wrong. Currently testing #8365 with `--user-config '{"foo":"bar"}'` but getting
```
02:44:49.597 [function-timer-thread-106-1] INFO org.apache.pulsar.functions.runtime.process.ProcessRuntime - Started process successfully
2020/12/01 02:44:49.603 log.go:46: [info] The default config file path is: conf/conf.yaml
2020/12/01 02:44:49.605 log.go:76: [error] unmarshal config content error:invalid character '\'' looking for beginning of value panic: config file is nil. goroutine 1 [running]:
github.com/apache/pulsar/pulsar-function-go/pf.newInstanceConf(0x62d40) /<home>/go/pkg/mod/github.com/apache/pulsar/pulsar-function-go@v0.0.0-20201130103018-9c28378aea7d/pf/instanceConf.go:49 +0x646 github.com/apache/pulsar/pulsar-function-go/pf.NewFuncContext(0x0)
/<home>/go/pkg/mod/github.com/apache/pulsar/pulsar-function-go@v0.0.0-20201130103018-9c28378aea7d/pf/context.go:44 +0x26 github.com/apache/pulsar/pulsar-function-go/pf.newGoInstance(0xbbdd40) /<home>/go/pkg/mod/github.com/apache/pulsar/pulsar-function-go@v0.0.0-20201130103018-9c28378aea7d/pf/instance.go:65 +0x37
github.com/apache/pulsar/pulsar-function-go/pf.Start(0xbbdd40, 0xce1918) /<home>/go/pkg/mod/github.com/apache/pulsar/pulsar-function-go@v0.0.0-20201130103018-9c28378aea7d/pf/function.go:170 +0x52
main.main() /<home>/go/src/scratch/pulsartest/cmd/function/main.go:62 +0x39
```
where `main.go:62` corresponds to:
```go
pf.Start(contextFunc)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] flowchartsman commented on issue #8216: Go SDK: GetUserConfMap() is not getting populated
Posted by GitBox <gi...@apache.org>.
flowchartsman commented on issue #8216:
URL: https://github.com/apache/pulsar/issues/8216#issuecomment-736224054
@vaihtovirta I have discovered a couple of issues. The first issue is that spurious single quotes (`'`) are getting added to the arguments to the pulsar function, causing it to fail. The offending line is located in `pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/RuntimeUtils.java`:
```java
args.add("'" + configContent + "'");
```
Should be:
```java
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org