You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/03/10 15:50:04 UTC
[GitHub] [pulsar] fmiguelez opened a new issue #9867: Script gen-yml-from-env.py does not work with array types
fmiguelez opened a new issue #9867:
URL: https://github.com/apache/pulsar/issues/9867
**Describe the bug**
Script `gen-yml-from-env.py` is typically used when separate function worker instances (e.g. on a Docker cluster) are configured. This script takes environment variables starting with `PF_` prefix and assigns their values to `conf/functions_worker.yml`.
Some properties in YAML file take the form of an array. If we set one of those properties with an environment variable, e.g. `PF_authenticationProviders` as we would do to configure the broker or proxy (in this case the variable would be `authenticationProviders` without the prefix) the script will produce an invalid YAML file that will fail with following error:
> Exception in thread "main" com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.util.HashSet<java.lang.Object>` out of VALUE_STRING token
> at [Source: (File); line: 7, column: 26] (through reference chain: org.apache.pulsar.functions.worker.WorkerConfig["authenticationProviders"])
> at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
> at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1468)
> at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1242)
> at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1148)
> at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.handleNonArray(StringCollectionDeserializer.java:274)
> at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:183)
> at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:173)
> at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:21)
> at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
> at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:293)
> at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:156)
> at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4482)
> at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3299)
> at org.apache.pulsar.functions.worker.WorkerConfig.load(WorkerConfig.java:491)
> at org.apache.pulsar.functions.worker.FunctionWorkerStarter.main(FunctionWorkerStarter.java:62)
We can define indicated variable `PF_authenticationProviders` taking value `org.apache.pulsar.broker.authentication.AuthenticationProviderTls` or `org.apache.pulsar.broker.authentication.AuthenticationProviderTls,org.apache.pulsar.broker.authentication.AuthenticationProviderAthenz`.
**To Reproduce**
We can execute following docker command to reproduce the issue:
```shell
docker run -e PF_authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls apachepulsar/pulsar:2.7.0 bash -c "bin/gen-yml-from-env.py conf/functions_worker.yml; bin/pulsar functions-worker;"
```
**Expected behavior**
Produced configuration file must be valid for function workers.
**Screenshots**
**Desktop (please complete the following information):**
- OS: Tested on Windows with Docker Desktop
**Additional context**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] murong00 commented on issue #9867: Script gen-yml-from-env.py does not work with array types
Posted by GitBox <gi...@apache.org>.
murong00 commented on issue #9867:
URL: https://github.com/apache/pulsar/issues/9867#issuecomment-796553408
You can try to use `[org.apache.pulsar.broker.authentication.AuthenticationProviderTls]` or `[org.apache.pulsar.broker.authentication.AuthenticationProviderTls,org.apache.pulsar.broker.authentication.AuthenticationProviderAthen]` to fetch values of array types.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie closed issue #9867: Script gen-yml-from-env.py does not work with array types
Posted by GitBox <gi...@apache.org>.
sijie closed issue #9867:
URL: https://github.com/apache/pulsar/issues/9867
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] fmiguelez commented on issue #9867: Script gen-yml-from-env.py does not work with array types
Posted by GitBox <gi...@apache.org>.
fmiguelez commented on issue #9867:
URL: https://github.com/apache/pulsar/issues/9867#issuecomment-796623650
> You can try to use `[org.apache.pulsar.broker.authentication.AuthenticationProviderTls]` or `[org.apache.pulsar.broker.authentication.AuthenticationProviderTls,org.apache.pulsar.broker.authentication.AuthenticationProviderAthen]` to fetch values of array types.
It will not work either. You can try your proposal yourself with (it fails with same error):
```shell
docker run -e "PF_authenticationProviders=[org.apache.pulsar.broker.authentication.AuthenticationProviderTls]" apachepulsar/pulsar:2.7.0 bash -c "bin/gen-yml-from-env.py conf/functions_worker.yml; bin/pulsar functions-worker;"
```
The problem is that they python script interprets the value as a string so final value you will see in yaml file will be:
```yaml
authenticationProviders: '[org.apache.pulsar.broker.authentication.AuthenticationProviderTls]'
```
I does not matter if you enclose the value in square brackets or not, final result will always be quoted (string) and thus provoke an exception on Java when trying to read it as a `Set<String>`.
I have some changes on the script so I when we finish testing it locally I will publish a pull request with the corrections to solve the issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org