You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/02/10 17:13:52 UTC
[GitHub] [pulsar-dotpulsar] oven-baked opened a new issue #68: Consuming a Message generated by a Python Pulsar Function puts Consumer in Faulted state.
oven-baked opened a new issue #68:
URL: https://github.com/apache/pulsar-dotpulsar/issues/68
I've been generating messages in a Python Pulsar function and spitting them out a receiving topic.
When I consume from that topic using the client app bundled with Pulsar, everything works okay and I can pull down and see my resulting messages.
If the DotPulsar consumer tries to pick up any of those messages from the function, it immediately enters a faulted state.
The exception message is "{"Specified argument was out of the range of valid values. (Parameter 'length')"}"
I've been able to pull the following stack trace from the library:
`System.ThrowHelper.ThrowStartOrEndArgumentValidationException(Int64 start)
at System.Buffers.ReadOnlySequence`1.Slice(Int64 start, Int64 length)
at DotPulsar.Internal.BatchHandler.Add(MessageIdData messageId, UInt32 redeliveryCount, MessageMetadata metadata, ReadOnlySequence`1 data) in C:\Users\X2012039\source\repos\pulsar-dotpulsar\src\DotPulsar\Internal\BatchHandler.cs:line 51
at DotPulsar.Internal.ConsumerChannel.<Receive>d__9.MoveNext() in C:\Users\X2012039\source\repos\pulsar-dotpulsar\src\DotPulsar\Internal\ConsumerChannel.cs:line 89
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
at DotPulsar.Internal.Consumer.<Receive>d__17.MoveNext() in C:\Users\X2012039\source\repos\pulsar-dotpulsar\src\DotPulsar\Internal\Consumer.cs:line 99
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
at DotPulsar.Internal.Executor.<Execute>d__9`1.MoveNext() in C:\Users\X2012039\source\repos\pulsar-dotpulsar\src\DotPulsar\Internal\Executor.cs:line 136`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] blankensteiner commented on issue #68: Consuming a Message generated by a Python Pulsar Function puts Consumer in Faulted state.
Posted by GitBox <gi...@apache.org>.
blankensteiner commented on issue #68:
URL: https://github.com/apache/pulsar-dotpulsar/issues/68#issuecomment-776893360
Hi @oven-baked
Thanks for the bug report, I'll look into this tomorrow.
Can you confirm that the Python function is batching messages?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] blankensteiner commented on issue #68: Consuming a Message generated by a Python Pulsar Function puts Consumer in Faulted state.
Posted by GitBox <gi...@apache.org>.
blankensteiner commented on issue #68:
URL: https://github.com/apache/pulsar-dotpulsar/issues/68#issuecomment-777440578
Thanks!
I see the problem.
Some time ago I checked if a message was batched or not by checking the NumOfMessagesInBatch field, which has a default value of '1'.
If it was '1' (no matter if it's the default value or explicitly set), then I assumed the message was not batched. However, this caused problems because Python, with batching enabled, would send a single message as a batched message. So, I changed the logic to check if the NumOfMessagesInBatch field was set or not (ignoring what it's set to), and if it was set then it's a batched message.
Now it seems that a message with NumOfMessagesInBatch set to '1' is generated from Python, but it's not a batched message.
Reverting this back to the original logic would fix the problem, but then I think we would break something else.
@sijie got any valuable insights here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] blankensteiner commented on issue #68: Consuming a Message generated by a Python Pulsar Function puts Consumer in Faulted state.
Posted by GitBox <gi...@apache.org>.
blankensteiner commented on issue #68:
URL: https://github.com/apache/pulsar-dotpulsar/issues/68#issuecomment-777706562
Hi @oven-baked
I had a quick look at Pulsar functions (mainly deployed via the YAML file) about 1½ year ago, but it's not something I know much about.
I would find it strange (and a bug) if compression could not be controlled (disabled or selecting from the other compression options). I think you have to ask in the Slack channel or mailing list or something like that :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] oven-baked commented on issue #68: Consuming a Message generated by a Python Pulsar Function puts Consumer in Faulted state.
Posted by GitBox <gi...@apache.org>.
oven-baked commented on issue #68:
URL: https://github.com/apache/pulsar-dotpulsar/issues/68#issuecomment-777300403
Hi @blankensteiner
Running Pulsar 2.7.0 and DotPulsar client is 0.10.1.
There has been nothing explicitly set around batching for the function. Is this something that needs to be tweaked for messages to safely consumed by the client? Apologies if this turns out to be configuration issues on my side!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] blankensteiner commented on issue #68: Consuming a Message generated by a Python Pulsar Function puts Consumer in Faulted state.
Posted by GitBox <gi...@apache.org>.
blankensteiner commented on issue #68:
URL: https://github.com/apache/pulsar-dotpulsar/issues/68#issuecomment-777258631
I tried producing some messages from a java client with batching enabled, creating batches containing 1 to multiple messages. I was able to read the messages just fine from DotPulsar.
Maybe the Python function is creating batched messages in a way that DotPulsar doesn't expect.
What version of DotPulsar are you using?
And what version of Pulsar are the functions running on?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] blankensteiner edited a comment on issue #68: Consuming a Message generated by a Python Pulsar Function puts Consumer in Faulted state.
Posted by GitBox <gi...@apache.org>.
blankensteiner edited a comment on issue #68:
URL: https://github.com/apache/pulsar-dotpulsar/issues/68#issuecomment-777459731
Ah.... looks like the Python function is compressing the message using LZ4.
Maybe that is why it can't extract the messages in the batched message because DotPulsar doesn't support compression (yet).
@oven-baked could you try to disable compression?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] blankensteiner commented on issue #68: Consuming a Message generated by a Python Pulsar Function puts Consumer in Faulted state.
Posted by GitBox <gi...@apache.org>.
blankensteiner commented on issue #68:
URL: https://github.com/apache/pulsar-dotpulsar/issues/68#issuecomment-782209329
Hi @oven-baked
I have just added support for LZ4, Snappy, Zlib, and Zstd compression. This will be available in the next release (0.11.0).
If you want to help test the bits in master, then feel free to reach out on slack :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] blankensteiner commented on issue #68: Consuming a Message generated by a Python Pulsar Function puts Consumer in Faulted state.
Posted by GitBox <gi...@apache.org>.
blankensteiner commented on issue #68:
URL: https://github.com/apache/pulsar-dotpulsar/issues/68#issuecomment-777348231
Hi @oven-baked
I suspect that this is a bug in DotPulsar.
Could you maybe provide some python code that will generate messages that make DotPulsar fail?
That would make it easier for me to debug.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] blankensteiner commented on issue #68: Consuming a Message generated by a Python Pulsar Function puts Consumer in Faulted state.
Posted by GitBox <gi...@apache.org>.
blankensteiner commented on issue #68:
URL: https://github.com/apache/pulsar-dotpulsar/issues/68#issuecomment-777459731
Ah.... looks like the Python function is compressing the message using LZ4.
Maybe that is why it can't extract the messages in the batched message because DotPulsar doesn't support compression (yet).
@oven-baked should you try to disable compression?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] oven-baked commented on issue #68: Consuming a Message generated by a Python Pulsar Function puts Consumer in Faulted state.
Posted by GitBox <gi...@apache.org>.
oven-baked commented on issue #68:
URL: https://github.com/apache/pulsar-dotpulsar/issues/68#issuecomment-777360419
Of course! We've been using this really simple function as supplied in the Pulsar documentation.
```
from pulsar import Function
class ExclamationFunction(Function):
def __init__(self):
pass
def process(self, input, context):
return input + '!'
```
And then installing it via:
`./pulsar-admin functions create --tenant public --namespace default --name MyFunction --py /home/oven-baked/apache-pulsar-2.7.0/bin/myfunction.py --classname myfunction.ExclamationFunction --inputs persistent://public/default/functioninputtopic --output persistent://public/default/myconsumertopic`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] oven-baked commented on issue #68: Consuming a Message generated by a Python Pulsar Function puts Consumer in Faulted state.
Posted by GitBox <gi...@apache.org>.
oven-baked commented on issue #68:
URL: https://github.com/apache/pulsar-dotpulsar/issues/68#issuecomment-777490393
Hey @blankensteiner, I've been rooting around the documentation and config files, but it doesn't _appear_ like there is any way to change compression being applied. I'm not even able to see the current compression algorithm in the function metadata.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] blankensteiner closed issue #68: Consuming a Message generated by a Python Pulsar Function puts Consumer in Faulted state.
Posted by GitBox <gi...@apache.org>.
blankensteiner closed issue #68:
URL: https://github.com/apache/pulsar-dotpulsar/issues/68
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org