kafka producer as an aws lambda function vulnerable to dropped messages

[Lee Carroll <le...@googlemail.com>]

Hi All,
I've been experimenting with a Kafka producer implemented as an AWS lambda
function and intial tests have shown potential issues with this approach.
I'm new to Kafka so I may have made some basics errors.

I've searched across the mailing list but nothing related is coming up so
hopefully this question is not an obvious duplicate.

In short is running kafka as a lambda producer a good idea? Or more
generally running a producer as a FAAS a good idea. I'm thinking possibly
not given the below experience.

My set up

I have a producer p which has a dependency of a configured producer.

public class WebEventProducer implements
RequestHandler<WebEventRequest, WebEventResponse> {

    private Producer<String, String> producer ;


These dependencies are configured using example set up during construction with

bootstrap.servers = one-of-my-brokers:9093
acks = all
retries = 0
batch.size = 16384
linger.ms = 1
timeout.ms = 1000
request.timeout.ms = 1000
metadata.fetch.timeout.ms = 2000
buffer.memory = 33554432
key.serializer = org.apache.kafka.common.serialization.StringSerializer
value.serializer = org.apache.kafka.common.serialization.StringSerializer


I async send messages and I do very little in the callback. Producer
#close or producer #flush

are not called.


What I observe:

Under load the producer sends messages very nicely thank you very
much. I'm interpreting this as the function container is persisted and
the handler

function is reused. The state held in the producer is reused and
nothing is lost. All good.

However if load stops I observe two things.

1. a message can be held in the producer buffer without being sent. In
order for the message to be sent further messages must be processed.

2. If subsequent messages are not sent the functional container gets
tore down after x amount of time and the message is lost.

*Now I know I'm cheating a little by having state in my
**dependencies** but in the main it works very well. However are the
above edge cases fatal. I'm thinking*

*that maybe I can get round this issue by calling #flush in the call
back but is this good practice? Also I simply don't have any ability
to call #close*

*as I don't appear to have a decent hook. Does this matter? Is kafka
tolerant of hung up producers, will they be cleaned up?*

*Lastly under very heavy load the FAAS container could spin up a 1000
containers all running an instance of my producer (that is extreme and
we'd never get their*

*but it is in theory possible). Is it okay to have hundreds of
producers all producing the same topic?*

*Really sorry for the long question and the rambling information...*


*Lee C*