You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Dave Peterson <da...@dspeterson.com> on 2016/06/11 22:24:54 UTC
Introducing Dory
Hello Kafka users,
Version 1.1.0 of Dory is now available. See
https://github.com/dspeterson/dory for details. Dory is the successor
to Bruce (https://github.com/tagged/bruce), a Kafka producer daemon I
created while working at if(we) (http://www.ifwe.co/). The code has
seen a number of improvements since its initial release in September
2014. The list of example clients for various programming languages
has also been extended. Dory maintains full backward compatibility
with Bruce, so existing users can easily switch.
The latest release adds support for receiving messages from clients by
UNIX domain stream socket or local TCP. Although UNIX domain
datagrams are still the preferred means of sending messages in most
cases, the option of using stream sockets facilitates sending messages
too large to fit in a single datagram. The local TCP option
facilitates adding support for clients written in programming
languages that do not provide easy access to UNIX domain sockets.
Dory's wiki page http://dory.wikidot.com/start contains a list of
ideas for additional features and other improvements. Community
feedback is welcomed and appreciated. If you have ideas for things
you would like to see in future releases, please add them to the list.
Also, please contribute code if you can afford the time.
Thanks,
Dave Peterson
Re: Introducing Dory
Posted by Dave Peterson <da...@dspeterson.com>.
There is an option for writing messages to a local log file, but this is
intended
purely for troubleshooting. Dory's intended purpose is to get messages off
the local machine as quickly and efficiently as possible, and let the Kafka
brokers handle the persistence.
When a client sends a message to Dory, there is no ACK from Dory back to
the client. An ACK isn't needed since Dory's clients are local, and the
operating system guarantees the reliability of the interprocess
communication. However, when Dory forwards the messages to Kafka, it
waits for an ACK and resends messages as necessary due to communication
failures or other errors. Once Dory has received a message from a
client, it
takes full responsibility for reliable delivery to Kafka. This
simplifies clients,
since sending messages becomes a simple one-way "fire it and forget it"
communication process.
Regards,
Dave
On 6/11/2016 7:50 PM, Tauzell, Dave wrote:
> Does Dory have the option to persist messages to local disk after the Dory ack but before sending to Kafka?
>
> Dave
>
>> On Jun 11, 2016, at 17:23, Dave Peterson <da...@dspeterson.com> wrote:
>>
>> Hello Kafka users,
>>
>> Version 1.1.0 of Dory is now available. See
>> https://github.com/dspeterson/dory for details. Dory is the successor
>> to Bruce (https://github.com/tagged/bruce), a Kafka producer daemon I
>> created while working at if(we) (http://www.ifwe.co/). The code has
>> seen a number of improvements since its initial release in September
>> 2014. The list of example clients for various programming languages
>> has also been extended. Dory maintains full backward compatibility
>> with Bruce, so existing users can easily switch.
>>
>> The latest release adds support for receiving messages from clients by
>> UNIX domain stream socket or local TCP. Although UNIX domain
>> datagrams are still the preferred means of sending messages in most
>> cases, the option of using stream sockets facilitates sending messages
>> too large to fit in a single datagram. The local TCP option
>> facilitates adding support for clients written in programming
>> languages that do not provide easy access to UNIX domain sockets.
>>
>> Dory's wiki page http://dory.wikidot.com/start contains a list of
>> ideas for additional features and other improvements. Community
>> feedback is welcomed and appreciated. If you have ideas for things
>> you would like to see in future releases, please add them to the list.
>> Also, please contribute code if you can afford the time.
>>
>>
>> Thanks,
>> Dave Peterson
>>
>>
> This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: Introducing Dory
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
Does Dory have the option to persist messages to local disk after the Dory ack but before sending to Kafka?
Dave
> On Jun 11, 2016, at 17:23, Dave Peterson <da...@dspeterson.com> wrote:
>
> Hello Kafka users,
>
> Version 1.1.0 of Dory is now available. See
> https://github.com/dspeterson/dory for details. Dory is the successor
> to Bruce (https://github.com/tagged/bruce), a Kafka producer daemon I
> created while working at if(we) (http://www.ifwe.co/). The code has
> seen a number of improvements since its initial release in September
> 2014. The list of example clients for various programming languages
> has also been extended. Dory maintains full backward compatibility
> with Bruce, so existing users can easily switch.
>
> The latest release adds support for receiving messages from clients by
> UNIX domain stream socket or local TCP. Although UNIX domain
> datagrams are still the preferred means of sending messages in most
> cases, the option of using stream sockets facilitates sending messages
> too large to fit in a single datagram. The local TCP option
> facilitates adding support for clients written in programming
> languages that do not provide easy access to UNIX domain sockets.
>
> Dory's wiki page http://dory.wikidot.com/start contains a list of
> ideas for additional features and other improvements. Community
> feedback is welcomed and appreciated. If you have ideas for things
> you would like to see in future releases, please add them to the list.
> Also, please contribute code if you can afford the time.
>
>
> Thanks,
> Dave Peterson
>
>
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: Introducing Dory
Posted by Dave Peterson <da...@dspeterson.com>.
Yes, the persistence feature could also be configurable on a per-topic
basis. If desired, persistence could then be limited only to certain
critical topics, allowing less critical data to avoid the overhead.
There could also be configuration options specifying when and how to
persist the data. Some options might be to fsync() the data to disk
immediately, at specified intervals, or just let the kernel decide when
to write the dirty pages to disk from its page cache.
Dave
-----Original Message-----
From: "Pete Wright" <pw...@rubiconproject.com>
Sent: Monday, June 13, 2016 11:22am
To: users@kafka.apache.org
Subject: Re: Introducing Dory
On Mon, Jun 13, 2016 at 11:45:13AM -0600, Jason J. W. Williams wrote:
> Hi Dave,
>
> Dory sounds very exciting. Without persistence its less useful for clients
> connected over a WAN, since if the WAN goes wonky you could build up quite
> a queue until it comes back.
>
I was thinking the same thing. My first thought was to see what the LOE
would be to implement some sort of spill-over process, where if the
preallocated memory segment is exhausted it could spool data to disk.
When connectivity to the brokers is back it could then de-spool data and
produce it back to the cluster. I could see this being something worth
persuing if you are handling data that is critical for financial
reasons (as opposed to data used for non-financial reporting or
metrics).
-pete
--
Pete Wright
Lead Systems Architect
Rubicon Project
pwright@rubiconproject.com
310.309.9298
Re: Introducing Dory
Posted by Pete Wright <pw...@rubiconproject.com>.
On Mon, Jun 13, 2016 at 11:45:13AM -0600, Jason J. W. Williams wrote:
> Hi Dave,
>
> Dory sounds very exciting. Without persistence its less useful for clients
> connected over a WAN, since if the WAN goes wonky you could build up quite
> a queue until it comes back.
>
I was thinking the same thing. My first thought was to see what the LOE
would be to implement some sort of spill-over process, where if the
preallocated memory segment is exhausted it could spool data to disk.
When connectivity to the brokers is back it could then de-spool data and
produce it back to the cluster. I could see this being something worth
persuing if you are handling data that is critical for financial
reasons (as opposed to data used for non-financial reporting or
metrics).
-pete
--
Pete Wright
Lead Systems Architect
Rubicon Project
pwright@rubiconproject.com
310.309.9298
Re: Introducing Dory
Posted by "Jason J. W. Williams" <ja...@gmail.com>.
Awesome! Thanks Dave! Right now we're using a chain of Apache Flume
instances to do this (since Flume doesn't support SSL->Kafka), but Dory
with persistence would let us simplify that a lot straight into Kafka from
Dory.
-J
On Mon, Jun 13, 2016 at 12:23 PM, Dave Peterson <da...@dspeterson.com> wrote:
> Hmm... interesting use case. I can see the usefulness of persistence in
> that scenario. Thanks very much for the feedback! I just added your
> idea to Dory's wiki page http://dory.wikidot.com . Progress at adding
> features has been slow lately, since I work on Dory in my limited spare
> time. But I welcome all the outside help I can get. If you or others
> are interested in working on adding a persistence feature, that would be
> great. Otherwise hopefully I'll get a chance to work on it myself at
> some point. Thanks again for the suggestion!
>
>
> Regards,
> Dave
>
>
> -----Original Message-----
> From: "Jason J. W. Williams" <ja...@gmail.com>
> Sent: Monday, June 13, 2016 10:45am
> To: users@kafka.apache.org
> Subject: Re: Introducing Dory
>
> Hi Dave,
>
> Dory sounds very exciting. Without persistence its less useful for clients
> connected over a WAN, since if the WAN goes wonky you could build up quite
> a queue until it comes back.
>
> -J
>
> On Mon, Jun 13, 2016 at 3:00 AM, Dave Peterson <da...@dspeterson.com>
> wrote:
>
> > Hi Arya,
> >
> > In the case of a kernel panic or other failure serious enough to bring
> > a machine down (hardware failure, power outage, etc.), message loss is
> > definitely possible, and not totally preventable regardless of what
> > precautions software takes to avoid data loss. To ensure high
> > throughput, Dory batches messages together before forwarding them to
> > Kafka. The amount of batching Dory performs is configurable on a
> > per-topic basis, and in practice will typically cause a delay of
> > between several hundred milliseconds and several seconds before
> > messages are forwarded to Kafka. Thus, if a machine on which Dory is
> > running crashes, you can expect to lose up to a few seconds of batched
> > messages depending on the batching configuration. Additionally, you
> > may lose some sent messages for which Dory has not yet received an ACK
> > from Kafka indicating successful receipt and persistence to disk.
> >
> > To narrow the window of possible message loss, one can set a very
> > short batching delay, or even completely disable batching for one or
> > more topics. However this is not recommended since it will have a
> > serious adverse affect on the performance of your Kafka cluster.
> > Another possibility would be for Dory to immediately persist messages
> > to local disk on receipt, even before they are batched, and then
> > delete them only after a non-error ACK has been received from Kafka.
> > This would greatly narrow, although not totally eliminate the window
> > for message loss. However it would add substantial complexity to
> > Dory, and impose a substantial disk I/O performance cost. An
> > important consideration is that when you write to a file, the kernel's
> > default action is to not immediately write the data to disk. It
> > buffers the data in its page cache for a period of time (typically up
> > to 30 seconds) to improve throughput. If the machine crashes, all of
> > this unwritten data is lost. You can get around this by calling
> > fsync() on a file descriptor or specifying O_DIRECT | O_SYNC when
> > opening a file, but this will degrade I/O performance. Dory avoids
> > persisting data to disk with the view that these downsides outweigh
> > the benefits of avoiding up to a few seconds of data loss when a
> > machine crashes. However a feature I intend to add (described later
> > in this email) will provide useful information about which messages
> > were successfully delivered to Kafka in the event of a machine crash.
> >
> > Dory takes great care to avoid message loss due to circumstances
> > other than machine crashes. When Dory starts up, it preallocates a
> > fixed amount of memory (configurable with the --msg_buffer_max N
> > command line option) for storing messages received from clients.
> > Dory holds on to each message it receives until one of the following
> > has occurred:
> >
> > - It receives an ACK from Kafka indicating that the message has
> > been successfully received and persisted to disk.
> >
> > - It receives an error ACK from Kafka that causes the message to
> > be immediately discarded. Certain error ACKs cause Dory to
> > immediately discard messages, and others cause it to pause,
> > update its metadata, and attempt redelivery based on the new
> > metadata. ACK error 2 (invalid message, i.e. CRC error) causes
> > Dory to immediately attempt redelivery without updating its
> > metadata. The actions Dory takes for various error ACK values
> > are documented here:
> >
> >
> > https://github.com/dspeterson/dory/blob/master/doc/design.md#dispatcher
> >
> > - As mentioned above, certain error ACKs will cause Dory to
> > attempt redelivery. Each time this occurs for a given message,
> > Dory increments a failed delivery attempt count on the message.
> > Once the count exceeds a certain value (specified by the
> > --max_failed_delivery_attempts N command line option), Dory
> > discards the message.
> >
> > - It has determined that the message is undeliverable due to a
> > problem such as the message being malformed (see
> >
> > https://github.com/dspeterson/dory/blob/master/doc/sending_messages.md
> > which documents Dory's input message format) or having an
> > invalid topic. In this case, the message is discarded.
> >
> > - It has exhausted its preallocated memory for storing messages.
> > This may occur in the case where a network-related outage or
> > serious problem with the Kafka cluster persists for an extended
> > period of time, preventing normal message delivery. In this
> > case, new messages received from clients are discarded.
> >
> > - Dory receives a shutdown signal (SIGTERM or SIGINT). In this
> > case, Dory immediately stops accepting new messages from
> > clients, and for a certain configurable period of time attempts
> > to deliver any pending messages. Once the time limit expires,
> > any remaining messages are lost. This type of message loss
> > can be avoided by not shutting Dory down until no more clients
> > are sending it messages, and Dory has been given enough time to
> > process all pending messages.
> >
> > A critical aspect of Dory's behavior is that every discard is tracked
> > along with the reason why the discard occurred. Dory provides a web
> > interface from which periodic discard reports and other status
> > information can be obtained. Thus, in the rare event when discards
> > occur, they can be tracked and recorded on a per-topic basis. The
> > intent is for monitoring infrastructure to periodically ask Dory for
> > a discard report, alert an administrator if message loss occurred, and
> > optionally store the discard reports in a database so that a queryable
> > history of data quality is available.
> >
> > A relevant feature I intend to add at some point is described here:
> >
> > http://dory.wikidot.com/wiki:commit-points
> >
> > This notion of a commit point can be useful in the event of a machine
> > crash. If monitoring infrastructure periodically querys Dory for
> > commit point information, then in the event of a machine crash, we
> > know that any resulting message loss is limited to messages with
> > timestamps > T, where T is the latest commit point.
> >
> > I hope this helps answer your questions, and that I haven't inundated
> > you with too much information. Let me know if you have more
> > questions.
> >
> >
> > Regards,
> > Dave
> >
> >
> > -----Original Message-----
> > From: "Arya Ketan" <ke...@gmail.com>
> > Sent: Sunday, June 12, 2016 11:52am
> > To: users@kafka.apache.org
> > Subject: Re: Introducing Dory
> >
> > Hi Dave,
> > Dory looks pretty interesting. I had a few further questions on it
> > a) How does Dory handle kernel panics?
> > b) What kind of message guarantees does dory provide and also if you can
> > share some design decisions taken to enable the guarantees whatever they
> > are.
> >
> > Thanks
> > Arya
> >
> > Arya
> >
> > On Sun, Jun 12, 2016 at 8:10 PM, Dave Peterson <da...@dspeterson.com>
> > wrote:
> >
> > > Thanks! Enjoy :-)
> > >
> > >
> > >
> > > On 6/12/2016 12:24 AM, Gwen Shapira wrote:
> > >
> > >> Dory is pretty cool (even though it is named after a somewhat dorky
> > >> fish). Thank you for sharing :)
> > >>
> > >> On Sun, Jun 12, 2016 at 1:24 AM, Dave Peterson <da...@dspeterson.com>
> > >> wrote:
> > >>
> > >>> Hello Kafka users,
> > >>>
> > >>> Version 1.1.0 of Dory is now available. See
> > >>> https://github.com/dspeterson/dory for details. Dory is the
> successor
> > >>> to Bruce (https://github.com/tagged/bruce), a Kafka producer daemon
> I
> > >>> created while working at if(we) (http://www.ifwe.co/). The code has
> > >>> seen a number of improvements since its initial release in September
> > >>> 2014. The list of example clients for various programming languages
> > >>> has also been extended. Dory maintains full backward compatibility
> > >>> with Bruce, so existing users can easily switch.
> > >>>
> > >>> The latest release adds support for receiving messages from clients
> by
> > >>> UNIX domain stream socket or local TCP. Although UNIX domain
> > >>> datagrams are still the preferred means of sending messages in most
> > >>> cases, the option of using stream sockets facilitates sending
> messages
> > >>> too large to fit in a single datagram. The local TCP option
> > >>> facilitates adding support for clients written in programming
> > >>> languages that do not provide easy access to UNIX domain sockets.
> > >>>
> > >>> Dory's wiki page http://dory.wikidot.com/start contains a list of
> > >>> ideas for additional features and other improvements. Community
> > >>> feedback is welcomed and appreciated. If you have ideas for things
> > >>> you would like to see in future releases, please add them to the
> list.
> > >>> Also, please contribute code if you can afford the time.
> > >>>
> > >>>
> > >>> Thanks,
> > >>> Dave Peterson
> > >>>
> > >>>
> > >>>
> > >
> >
> >
> >
>
>
Re: Introducing Dory
Posted by Dave Peterson <da...@dspeterson.com>.
Hmm... interesting use case. I can see the usefulness of persistence in
that scenario. Thanks very much for the feedback! I just added your
idea to Dory's wiki page http://dory.wikidot.com . Progress at adding
features has been slow lately, since I work on Dory in my limited spare
time. But I welcome all the outside help I can get. If you or others
are interested in working on adding a persistence feature, that would be
great. Otherwise hopefully I'll get a chance to work on it myself at
some point. Thanks again for the suggestion!
Regards,
Dave
-----Original Message-----
From: "Jason J. W. Williams" <ja...@gmail.com>
Sent: Monday, June 13, 2016 10:45am
To: users@kafka.apache.org
Subject: Re: Introducing Dory
Hi Dave,
Dory sounds very exciting. Without persistence its less useful for clients
connected over a WAN, since if the WAN goes wonky you could build up quite
a queue until it comes back.
-J
On Mon, Jun 13, 2016 at 3:00 AM, Dave Peterson <da...@dspeterson.com> wrote:
> Hi Arya,
>
> In the case of a kernel panic or other failure serious enough to bring
> a machine down (hardware failure, power outage, etc.), message loss is
> definitely possible, and not totally preventable regardless of what
> precautions software takes to avoid data loss. To ensure high
> throughput, Dory batches messages together before forwarding them to
> Kafka. The amount of batching Dory performs is configurable on a
> per-topic basis, and in practice will typically cause a delay of
> between several hundred milliseconds and several seconds before
> messages are forwarded to Kafka. Thus, if a machine on which Dory is
> running crashes, you can expect to lose up to a few seconds of batched
> messages depending on the batching configuration. Additionally, you
> may lose some sent messages for which Dory has not yet received an ACK
> from Kafka indicating successful receipt and persistence to disk.
>
> To narrow the window of possible message loss, one can set a very
> short batching delay, or even completely disable batching for one or
> more topics. However this is not recommended since it will have a
> serious adverse affect on the performance of your Kafka cluster.
> Another possibility would be for Dory to immediately persist messages
> to local disk on receipt, even before they are batched, and then
> delete them only after a non-error ACK has been received from Kafka.
> This would greatly narrow, although not totally eliminate the window
> for message loss. However it would add substantial complexity to
> Dory, and impose a substantial disk I/O performance cost. An
> important consideration is that when you write to a file, the kernel's
> default action is to not immediately write the data to disk. It
> buffers the data in its page cache for a period of time (typically up
> to 30 seconds) to improve throughput. If the machine crashes, all of
> this unwritten data is lost. You can get around this by calling
> fsync() on a file descriptor or specifying O_DIRECT | O_SYNC when
> opening a file, but this will degrade I/O performance. Dory avoids
> persisting data to disk with the view that these downsides outweigh
> the benefits of avoiding up to a few seconds of data loss when a
> machine crashes. However a feature I intend to add (described later
> in this email) will provide useful information about which messages
> were successfully delivered to Kafka in the event of a machine crash.
>
> Dory takes great care to avoid message loss due to circumstances
> other than machine crashes. When Dory starts up, it preallocates a
> fixed amount of memory (configurable with the --msg_buffer_max N
> command line option) for storing messages received from clients.
> Dory holds on to each message it receives until one of the following
> has occurred:
>
> - It receives an ACK from Kafka indicating that the message has
> been successfully received and persisted to disk.
>
> - It receives an error ACK from Kafka that causes the message to
> be immediately discarded. Certain error ACKs cause Dory to
> immediately discard messages, and others cause it to pause,
> update its metadata, and attempt redelivery based on the new
> metadata. ACK error 2 (invalid message, i.e. CRC error) causes
> Dory to immediately attempt redelivery without updating its
> metadata. The actions Dory takes for various error ACK values
> are documented here:
>
>
> https://github.com/dspeterson/dory/blob/master/doc/design.md#dispatcher
>
> - As mentioned above, certain error ACKs will cause Dory to
> attempt redelivery. Each time this occurs for a given message,
> Dory increments a failed delivery attempt count on the message.
> Once the count exceeds a certain value (specified by the
> --max_failed_delivery_attempts N command line option), Dory
> discards the message.
>
> - It has determined that the message is undeliverable due to a
> problem such as the message being malformed (see
>
> https://github.com/dspeterson/dory/blob/master/doc/sending_messages.md
> which documents Dory's input message format) or having an
> invalid topic. In this case, the message is discarded.
>
> - It has exhausted its preallocated memory for storing messages.
> This may occur in the case where a network-related outage or
> serious problem with the Kafka cluster persists for an extended
> period of time, preventing normal message delivery. In this
> case, new messages received from clients are discarded.
>
> - Dory receives a shutdown signal (SIGTERM or SIGINT). In this
> case, Dory immediately stops accepting new messages from
> clients, and for a certain configurable period of time attempts
> to deliver any pending messages. Once the time limit expires,
> any remaining messages are lost. This type of message loss
> can be avoided by not shutting Dory down until no more clients
> are sending it messages, and Dory has been given enough time to
> process all pending messages.
>
> A critical aspect of Dory's behavior is that every discard is tracked
> along with the reason why the discard occurred. Dory provides a web
> interface from which periodic discard reports and other status
> information can be obtained. Thus, in the rare event when discards
> occur, they can be tracked and recorded on a per-topic basis. The
> intent is for monitoring infrastructure to periodically ask Dory for
> a discard report, alert an administrator if message loss occurred, and
> optionally store the discard reports in a database so that a queryable
> history of data quality is available.
>
> A relevant feature I intend to add at some point is described here:
>
> http://dory.wikidot.com/wiki:commit-points
>
> This notion of a commit point can be useful in the event of a machine
> crash. If monitoring infrastructure periodically querys Dory for
> commit point information, then in the event of a machine crash, we
> know that any resulting message loss is limited to messages with
> timestamps > T, where T is the latest commit point.
>
> I hope this helps answer your questions, and that I haven't inundated
> you with too much information. Let me know if you have more
> questions.
>
>
> Regards,
> Dave
>
>
> -----Original Message-----
> From: "Arya Ketan" <ke...@gmail.com>
> Sent: Sunday, June 12, 2016 11:52am
> To: users@kafka.apache.org
> Subject: Re: Introducing Dory
>
> Hi Dave,
> Dory looks pretty interesting. I had a few further questions on it
> a) How does Dory handle kernel panics?
> b) What kind of message guarantees does dory provide and also if you can
> share some design decisions taken to enable the guarantees whatever they
> are.
>
> Thanks
> Arya
>
> Arya
>
> On Sun, Jun 12, 2016 at 8:10 PM, Dave Peterson <da...@dspeterson.com>
> wrote:
>
> > Thanks! Enjoy :-)
> >
> >
> >
> > On 6/12/2016 12:24 AM, Gwen Shapira wrote:
> >
> >> Dory is pretty cool (even though it is named after a somewhat dorky
> >> fish). Thank you for sharing :)
> >>
> >> On Sun, Jun 12, 2016 at 1:24 AM, Dave Peterson <da...@dspeterson.com>
> >> wrote:
> >>
> >>> Hello Kafka users,
> >>>
> >>> Version 1.1.0 of Dory is now available. See
> >>> https://github.com/dspeterson/dory for details. Dory is the successor
> >>> to Bruce (https://github.com/tagged/bruce), a Kafka producer daemon I
> >>> created while working at if(we) (http://www.ifwe.co/). The code has
> >>> seen a number of improvements since its initial release in September
> >>> 2014. The list of example clients for various programming languages
> >>> has also been extended. Dory maintains full backward compatibility
> >>> with Bruce, so existing users can easily switch.
> >>>
> >>> The latest release adds support for receiving messages from clients by
> >>> UNIX domain stream socket or local TCP. Although UNIX domain
> >>> datagrams are still the preferred means of sending messages in most
> >>> cases, the option of using stream sockets facilitates sending messages
> >>> too large to fit in a single datagram. The local TCP option
> >>> facilitates adding support for clients written in programming
> >>> languages that do not provide easy access to UNIX domain sockets.
> >>>
> >>> Dory's wiki page http://dory.wikidot.com/start contains a list of
> >>> ideas for additional features and other improvements. Community
> >>> feedback is welcomed and appreciated. If you have ideas for things
> >>> you would like to see in future releases, please add them to the list.
> >>> Also, please contribute code if you can afford the time.
> >>>
> >>>
> >>> Thanks,
> >>> Dave Peterson
> >>>
> >>>
> >>>
> >
>
>
>
Re: Introducing Dory
Posted by "Jason J. W. Williams" <ja...@gmail.com>.
Hi Dave,
Dory sounds very exciting. Without persistence its less useful for clients
connected over a WAN, since if the WAN goes wonky you could build up quite
a queue until it comes back.
-J
On Mon, Jun 13, 2016 at 3:00 AM, Dave Peterson <da...@dspeterson.com> wrote:
> Hi Arya,
>
> In the case of a kernel panic or other failure serious enough to bring
> a machine down (hardware failure, power outage, etc.), message loss is
> definitely possible, and not totally preventable regardless of what
> precautions software takes to avoid data loss. To ensure high
> throughput, Dory batches messages together before forwarding them to
> Kafka. The amount of batching Dory performs is configurable on a
> per-topic basis, and in practice will typically cause a delay of
> between several hundred milliseconds and several seconds before
> messages are forwarded to Kafka. Thus, if a machine on which Dory is
> running crashes, you can expect to lose up to a few seconds of batched
> messages depending on the batching configuration. Additionally, you
> may lose some sent messages for which Dory has not yet received an ACK
> from Kafka indicating successful receipt and persistence to disk.
>
> To narrow the window of possible message loss, one can set a very
> short batching delay, or even completely disable batching for one or
> more topics. However this is not recommended since it will have a
> serious adverse affect on the performance of your Kafka cluster.
> Another possibility would be for Dory to immediately persist messages
> to local disk on receipt, even before they are batched, and then
> delete them only after a non-error ACK has been received from Kafka.
> This would greatly narrow, although not totally eliminate the window
> for message loss. However it would add substantial complexity to
> Dory, and impose a substantial disk I/O performance cost. An
> important consideration is that when you write to a file, the kernel's
> default action is to not immediately write the data to disk. It
> buffers the data in its page cache for a period of time (typically up
> to 30 seconds) to improve throughput. If the machine crashes, all of
> this unwritten data is lost. You can get around this by calling
> fsync() on a file descriptor or specifying O_DIRECT | O_SYNC when
> opening a file, but this will degrade I/O performance. Dory avoids
> persisting data to disk with the view that these downsides outweigh
> the benefits of avoiding up to a few seconds of data loss when a
> machine crashes. However a feature I intend to add (described later
> in this email) will provide useful information about which messages
> were successfully delivered to Kafka in the event of a machine crash.
>
> Dory takes great care to avoid message loss due to circumstances
> other than machine crashes. When Dory starts up, it preallocates a
> fixed amount of memory (configurable with the --msg_buffer_max N
> command line option) for storing messages received from clients.
> Dory holds on to each message it receives until one of the following
> has occurred:
>
> - It receives an ACK from Kafka indicating that the message has
> been successfully received and persisted to disk.
>
> - It receives an error ACK from Kafka that causes the message to
> be immediately discarded. Certain error ACKs cause Dory to
> immediately discard messages, and others cause it to pause,
> update its metadata, and attempt redelivery based on the new
> metadata. ACK error 2 (invalid message, i.e. CRC error) causes
> Dory to immediately attempt redelivery without updating its
> metadata. The actions Dory takes for various error ACK values
> are documented here:
>
>
> https://github.com/dspeterson/dory/blob/master/doc/design.md#dispatcher
>
> - As mentioned above, certain error ACKs will cause Dory to
> attempt redelivery. Each time this occurs for a given message,
> Dory increments a failed delivery attempt count on the message.
> Once the count exceeds a certain value (specified by the
> --max_failed_delivery_attempts N command line option), Dory
> discards the message.
>
> - It has determined that the message is undeliverable due to a
> problem such as the message being malformed (see
>
> https://github.com/dspeterson/dory/blob/master/doc/sending_messages.md
> which documents Dory's input message format) or having an
> invalid topic. In this case, the message is discarded.
>
> - It has exhausted its preallocated memory for storing messages.
> This may occur in the case where a network-related outage or
> serious problem with the Kafka cluster persists for an extended
> period of time, preventing normal message delivery. In this
> case, new messages received from clients are discarded.
>
> - Dory receives a shutdown signal (SIGTERM or SIGINT). In this
> case, Dory immediately stops accepting new messages from
> clients, and for a certain configurable period of time attempts
> to deliver any pending messages. Once the time limit expires,
> any remaining messages are lost. This type of message loss
> can be avoided by not shutting Dory down until no more clients
> are sending it messages, and Dory has been given enough time to
> process all pending messages.
>
> A critical aspect of Dory's behavior is that every discard is tracked
> along with the reason why the discard occurred. Dory provides a web
> interface from which periodic discard reports and other status
> information can be obtained. Thus, in the rare event when discards
> occur, they can be tracked and recorded on a per-topic basis. The
> intent is for monitoring infrastructure to periodically ask Dory for
> a discard report, alert an administrator if message loss occurred, and
> optionally store the discard reports in a database so that a queryable
> history of data quality is available.
>
> A relevant feature I intend to add at some point is described here:
>
> http://dory.wikidot.com/wiki:commit-points
>
> This notion of a commit point can be useful in the event of a machine
> crash. If monitoring infrastructure periodically querys Dory for
> commit point information, then in the event of a machine crash, we
> know that any resulting message loss is limited to messages with
> timestamps > T, where T is the latest commit point.
>
> I hope this helps answer your questions, and that I haven't inundated
> you with too much information. Let me know if you have more
> questions.
>
>
> Regards,
> Dave
>
>
> -----Original Message-----
> From: "Arya Ketan" <ke...@gmail.com>
> Sent: Sunday, June 12, 2016 11:52am
> To: users@kafka.apache.org
> Subject: Re: Introducing Dory
>
> Hi Dave,
> Dory looks pretty interesting. I had a few further questions on it
> a) How does Dory handle kernel panics?
> b) What kind of message guarantees does dory provide and also if you can
> share some design decisions taken to enable the guarantees whatever they
> are.
>
> Thanks
> Arya
>
> Arya
>
> On Sun, Jun 12, 2016 at 8:10 PM, Dave Peterson <da...@dspeterson.com>
> wrote:
>
> > Thanks! Enjoy :-)
> >
> >
> >
> > On 6/12/2016 12:24 AM, Gwen Shapira wrote:
> >
> >> Dory is pretty cool (even though it is named after a somewhat dorky
> >> fish). Thank you for sharing :)
> >>
> >> On Sun, Jun 12, 2016 at 1:24 AM, Dave Peterson <da...@dspeterson.com>
> >> wrote:
> >>
> >>> Hello Kafka users,
> >>>
> >>> Version 1.1.0 of Dory is now available. See
> >>> https://github.com/dspeterson/dory for details. Dory is the successor
> >>> to Bruce (https://github.com/tagged/bruce), a Kafka producer daemon I
> >>> created while working at if(we) (http://www.ifwe.co/). The code has
> >>> seen a number of improvements since its initial release in September
> >>> 2014. The list of example clients for various programming languages
> >>> has also been extended. Dory maintains full backward compatibility
> >>> with Bruce, so existing users can easily switch.
> >>>
> >>> The latest release adds support for receiving messages from clients by
> >>> UNIX domain stream socket or local TCP. Although UNIX domain
> >>> datagrams are still the preferred means of sending messages in most
> >>> cases, the option of using stream sockets facilitates sending messages
> >>> too large to fit in a single datagram. The local TCP option
> >>> facilitates adding support for clients written in programming
> >>> languages that do not provide easy access to UNIX domain sockets.
> >>>
> >>> Dory's wiki page http://dory.wikidot.com/start contains a list of
> >>> ideas for additional features and other improvements. Community
> >>> feedback is welcomed and appreciated. If you have ideas for things
> >>> you would like to see in future releases, please add them to the list.
> >>> Also, please contribute code if you can afford the time.
> >>>
> >>>
> >>> Thanks,
> >>> Dave Peterson
> >>>
> >>>
> >>>
> >
>
>
>
Re: Introducing Dory
Posted by Dave Peterson <da...@dspeterson.com>.
Hi Arya,
In the case of a kernel panic or other failure serious enough to bring
a machine down (hardware failure, power outage, etc.), message loss is
definitely possible, and not totally preventable regardless of what
precautions software takes to avoid data loss. To ensure high
throughput, Dory batches messages together before forwarding them to
Kafka. The amount of batching Dory performs is configurable on a
per-topic basis, and in practice will typically cause a delay of
between several hundred milliseconds and several seconds before
messages are forwarded to Kafka. Thus, if a machine on which Dory is
running crashes, you can expect to lose up to a few seconds of batched
messages depending on the batching configuration. Additionally, you
may lose some sent messages for which Dory has not yet received an ACK
from Kafka indicating successful receipt and persistence to disk.
To narrow the window of possible message loss, one can set a very
short batching delay, or even completely disable batching for one or
more topics. However this is not recommended since it will have a
serious adverse affect on the performance of your Kafka cluster.
Another possibility would be for Dory to immediately persist messages
to local disk on receipt, even before they are batched, and then
delete them only after a non-error ACK has been received from Kafka.
This would greatly narrow, although not totally eliminate the window
for message loss. However it would add substantial complexity to
Dory, and impose a substantial disk I/O performance cost. An
important consideration is that when you write to a file, the kernel's
default action is to not immediately write the data to disk. It
buffers the data in its page cache for a period of time (typically up
to 30 seconds) to improve throughput. If the machine crashes, all of
this unwritten data is lost. You can get around this by calling
fsync() on a file descriptor or specifying O_DIRECT | O_SYNC when
opening a file, but this will degrade I/O performance. Dory avoids
persisting data to disk with the view that these downsides outweigh
the benefits of avoiding up to a few seconds of data loss when a
machine crashes. However a feature I intend to add (described later
in this email) will provide useful information about which messages
were successfully delivered to Kafka in the event of a machine crash.
Dory takes great care to avoid message loss due to circumstances
other than machine crashes. When Dory starts up, it preallocates a
fixed amount of memory (configurable with the --msg_buffer_max N
command line option) for storing messages received from clients.
Dory holds on to each message it receives until one of the following
has occurred:
- It receives an ACK from Kafka indicating that the message has
been successfully received and persisted to disk.
- It receives an error ACK from Kafka that causes the message to
be immediately discarded. Certain error ACKs cause Dory to
immediately discard messages, and others cause it to pause,
update its metadata, and attempt redelivery based on the new
metadata. ACK error 2 (invalid message, i.e. CRC error) causes
Dory to immediately attempt redelivery without updating its
metadata. The actions Dory takes for various error ACK values
are documented here:
https://github.com/dspeterson/dory/blob/master/doc/design.md#dispatcher
- As mentioned above, certain error ACKs will cause Dory to
attempt redelivery. Each time this occurs for a given message,
Dory increments a failed delivery attempt count on the message.
Once the count exceeds a certain value (specified by the
--max_failed_delivery_attempts N command line option), Dory
discards the message.
- It has determined that the message is undeliverable due to a
problem such as the message being malformed (see
https://github.com/dspeterson/dory/blob/master/doc/sending_messages.md
which documents Dory's input message format) or having an
invalid topic. In this case, the message is discarded.
- It has exhausted its preallocated memory for storing messages.
This may occur in the case where a network-related outage or
serious problem with the Kafka cluster persists for an extended
period of time, preventing normal message delivery. In this
case, new messages received from clients are discarded.
- Dory receives a shutdown signal (SIGTERM or SIGINT). In this
case, Dory immediately stops accepting new messages from
clients, and for a certain configurable period of time attempts
to deliver any pending messages. Once the time limit expires,
any remaining messages are lost. This type of message loss
can be avoided by not shutting Dory down until no more clients
are sending it messages, and Dory has been given enough time to
process all pending messages.
A critical aspect of Dory's behavior is that every discard is tracked
along with the reason why the discard occurred. Dory provides a web
interface from which periodic discard reports and other status
information can be obtained. Thus, in the rare event when discards
occur, they can be tracked and recorded on a per-topic basis. The
intent is for monitoring infrastructure to periodically ask Dory for
a discard report, alert an administrator if message loss occurred, and
optionally store the discard reports in a database so that a queryable
history of data quality is available.
A relevant feature I intend to add at some point is described here:
http://dory.wikidot.com/wiki:commit-points
This notion of a commit point can be useful in the event of a machine
crash. If monitoring infrastructure periodically querys Dory for
commit point information, then in the event of a machine crash, we
know that any resulting message loss is limited to messages with
timestamps > T, where T is the latest commit point.
I hope this helps answer your questions, and that I haven't inundated
you with too much information. Let me know if you have more
questions.
Regards,
Dave
-----Original Message-----
From: "Arya Ketan" <ke...@gmail.com>
Sent: Sunday, June 12, 2016 11:52am
To: users@kafka.apache.org
Subject: Re: Introducing Dory
Hi Dave,
Dory looks pretty interesting. I had a few further questions on it
a) How does Dory handle kernel panics?
b) What kind of message guarantees does dory provide and also if you can
share some design decisions taken to enable the guarantees whatever they
are.
Thanks
Arya
Arya
On Sun, Jun 12, 2016 at 8:10 PM, Dave Peterson <da...@dspeterson.com> wrote:
> Thanks! Enjoy :-)
>
>
>
> On 6/12/2016 12:24 AM, Gwen Shapira wrote:
>
>> Dory is pretty cool (even though it is named after a somewhat dorky
>> fish). Thank you for sharing :)
>>
>> On Sun, Jun 12, 2016 at 1:24 AM, Dave Peterson <da...@dspeterson.com>
>> wrote:
>>
>>> Hello Kafka users,
>>>
>>> Version 1.1.0 of Dory is now available. See
>>> https://github.com/dspeterson/dory for details. Dory is the successor
>>> to Bruce (https://github.com/tagged/bruce), a Kafka producer daemon I
>>> created while working at if(we) (http://www.ifwe.co/). The code has
>>> seen a number of improvements since its initial release in September
>>> 2014. The list of example clients for various programming languages
>>> has also been extended. Dory maintains full backward compatibility
>>> with Bruce, so existing users can easily switch.
>>>
>>> The latest release adds support for receiving messages from clients by
>>> UNIX domain stream socket or local TCP. Although UNIX domain
>>> datagrams are still the preferred means of sending messages in most
>>> cases, the option of using stream sockets facilitates sending messages
>>> too large to fit in a single datagram. The local TCP option
>>> facilitates adding support for clients written in programming
>>> languages that do not provide easy access to UNIX domain sockets.
>>>
>>> Dory's wiki page http://dory.wikidot.com/start contains a list of
>>> ideas for additional features and other improvements. Community
>>> feedback is welcomed and appreciated. If you have ideas for things
>>> you would like to see in future releases, please add them to the list.
>>> Also, please contribute code if you can afford the time.
>>>
>>>
>>> Thanks,
>>> Dave Peterson
>>>
>>>
>>>
>
Re: Introducing Dory
Posted by Arya Ketan <ke...@gmail.com>.
Hi Dave,
Dory looks pretty interesting. I had a few further questions on it
a) How does Dory handle kernel panics?
b) What kind of message guarantees does dory provide and also if you can
share some design decisions taken to enable the guarantees whatever they
are.
Thanks
Arya
Arya
On Sun, Jun 12, 2016 at 8:10 PM, Dave Peterson <da...@dspeterson.com> wrote:
> Thanks! Enjoy :-)
>
>
>
> On 6/12/2016 12:24 AM, Gwen Shapira wrote:
>
>> Dory is pretty cool (even though it is named after a somewhat dorky
>> fish). Thank you for sharing :)
>>
>> On Sun, Jun 12, 2016 at 1:24 AM, Dave Peterson <da...@dspeterson.com>
>> wrote:
>>
>>> Hello Kafka users,
>>>
>>> Version 1.1.0 of Dory is now available. See
>>> https://github.com/dspeterson/dory for details. Dory is the successor
>>> to Bruce (https://github.com/tagged/bruce), a Kafka producer daemon I
>>> created while working at if(we) (http://www.ifwe.co/). The code has
>>> seen a number of improvements since its initial release in September
>>> 2014. The list of example clients for various programming languages
>>> has also been extended. Dory maintains full backward compatibility
>>> with Bruce, so existing users can easily switch.
>>>
>>> The latest release adds support for receiving messages from clients by
>>> UNIX domain stream socket or local TCP. Although UNIX domain
>>> datagrams are still the preferred means of sending messages in most
>>> cases, the option of using stream sockets facilitates sending messages
>>> too large to fit in a single datagram. The local TCP option
>>> facilitates adding support for clients written in programming
>>> languages that do not provide easy access to UNIX domain sockets.
>>>
>>> Dory's wiki page http://dory.wikidot.com/start contains a list of
>>> ideas for additional features and other improvements. Community
>>> feedback is welcomed and appreciated. If you have ideas for things
>>> you would like to see in future releases, please add them to the list.
>>> Also, please contribute code if you can afford the time.
>>>
>>>
>>> Thanks,
>>> Dave Peterson
>>>
>>>
>>>
>
Re: Introducing Dory
Posted by Dave Peterson <da...@dspeterson.com>.
Thanks! Enjoy :-)
On 6/12/2016 12:24 AM, Gwen Shapira wrote:
> Dory is pretty cool (even though it is named after a somewhat dorky
> fish). Thank you for sharing :)
>
> On Sun, Jun 12, 2016 at 1:24 AM, Dave Peterson <da...@dspeterson.com> wrote:
>> Hello Kafka users,
>>
>> Version 1.1.0 of Dory is now available. See
>> https://github.com/dspeterson/dory for details. Dory is the successor
>> to Bruce (https://github.com/tagged/bruce), a Kafka producer daemon I
>> created while working at if(we) (http://www.ifwe.co/). The code has
>> seen a number of improvements since its initial release in September
>> 2014. The list of example clients for various programming languages
>> has also been extended. Dory maintains full backward compatibility
>> with Bruce, so existing users can easily switch.
>>
>> The latest release adds support for receiving messages from clients by
>> UNIX domain stream socket or local TCP. Although UNIX domain
>> datagrams are still the preferred means of sending messages in most
>> cases, the option of using stream sockets facilitates sending messages
>> too large to fit in a single datagram. The local TCP option
>> facilitates adding support for clients written in programming
>> languages that do not provide easy access to UNIX domain sockets.
>>
>> Dory's wiki page http://dory.wikidot.com/start contains a list of
>> ideas for additional features and other improvements. Community
>> feedback is welcomed and appreciated. If you have ideas for things
>> you would like to see in future releases, please add them to the list.
>> Also, please contribute code if you can afford the time.
>>
>>
>> Thanks,
>> Dave Peterson
>>
>>
Re: Introducing Dory
Posted by Gwen Shapira <gw...@confluent.io>.
Dory is pretty cool (even though it is named after a somewhat dorky
fish). Thank you for sharing :)
On Sun, Jun 12, 2016 at 1:24 AM, Dave Peterson <da...@dspeterson.com> wrote:
> Hello Kafka users,
>
> Version 1.1.0 of Dory is now available. See
> https://github.com/dspeterson/dory for details. Dory is the successor
> to Bruce (https://github.com/tagged/bruce), a Kafka producer daemon I
> created while working at if(we) (http://www.ifwe.co/). The code has
> seen a number of improvements since its initial release in September
> 2014. The list of example clients for various programming languages
> has also been extended. Dory maintains full backward compatibility
> with Bruce, so existing users can easily switch.
>
> The latest release adds support for receiving messages from clients by
> UNIX domain stream socket or local TCP. Although UNIX domain
> datagrams are still the preferred means of sending messages in most
> cases, the option of using stream sockets facilitates sending messages
> too large to fit in a single datagram. The local TCP option
> facilitates adding support for clients written in programming
> languages that do not provide easy access to UNIX domain sockets.
>
> Dory's wiki page http://dory.wikidot.com/start contains a list of
> ideas for additional features and other improvements. Community
> feedback is welcomed and appreciated. If you have ideas for things
> you would like to see in future releases, please add them to the list.
> Also, please contribute code if you can afford the time.
>
>
> Thanks,
> Dave Peterson
>
>