You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jay Kreps <ja...@gmail.com> on 2014/03/05 22:09:57 UTC
Documentation for the upcoming 0.8.1 release
Hey guys,
I took a stab at updating the docs for the 0.8.1 release. In particular, I
added a section on log compaction:
http://kafka.apache.org/081/documentation.html#compaction
I also updated the configs. This is all under 081, I will flip this over to
be the main documentation when 0.8.1 is released but you can see it now at
http://kafka.apache.org/081/documentation.html#compaction
Any comments on improvements we could make would be much appreciated!
-Jay
RE: Documentation for the upcoming 0.8.1 release
Posted by Vjeran Marcinko <vj...@email.t-com.hr>.
Well, what I'm saying is that message publisher doesn't know what consumers
are interested in, and sometimes they're interested in type of action
executed (event message that carries info about what command was executed),
and sometimes just in state change (what state fields are updated).
Eg. "activeCustomer" command, when processed successfully, can publish
"CustomerActivatedEvent" (information about type of command processed, no
mention about any customer field here)
and
CustomerUpdatedEvent (state change event, so this message complete new
state after the action has been processed)
In other words, these message correspond directly to difference between
business and DAO layer. Here is example of this service code:
public class CustomerService {
private CustomerDao customerDao;
public void activateCustomer(Long id) { // this has business
information - "event" message carries this
Customer customer = customerDao.findById(id);
customer.setActive(true);
customerDao.update(customer); // this has CRUD (state)
information - "state change" message carries this
}
}
The second message carries info about new state that primarily interests
consumers that only want to replicate state - eg. cache node, slave node
etc... For these messages, topic log compaction makes sense. But these
messages carry no info about what caused this state changes.
Whereas first type of message can interest other consumers, maybe some
real-time service that keeps track of frequency of customer activation (for
whatever reason). Of course, one can argue that its best to bundle together
all this info (business + state) into one message, but some actions are more
complex than this example, and cause multiple entities to be changed during
one transaction, thus the message would need to carry multiple state change
messages in itself for each entity that was changed during transaction,
which complicates things substantially.
It seems to me that most of examples found around for event-sourced systems
are with simple use cases, such as above.
Regards,
Vjeran
-----Original Message-----
From: Jay Kreps [mailto:jay.kreps@gmail.com]
Sent: Thursday, March 06, 2014 5:05 PM
To: users@kafka.apache.org
Subject: Re: Documentation for the upcoming 0.8.1 release
I think I understand what you are saying. I think you are saying that
perhaps you could break up a given problem into a kind of "system state"
and the actual data rows? This is not something we have had a need to do...
-Jay
On Thu, Mar 6, 2014 at 12:26 AM, Vjeran Marcinko <
vjeran.marcinko@email.t-com.hr> wrote:
> Thanx Jay,
>
> Somewhat related to log compaction....
>
> Did you have a need at LinkedIn where application should publish 2
> messages on separate topics for each action executed, one to "event
> data" topic demarking signalizing executed action, and another one to
"state change"
> topic which ultimately is good for state replication and can be
> compacted after period of time? This state change messages should
> maybe even contain reference to event message to be able to see the cause
of state change...
>
> Regards,
> Vjeran
>
> -----Original Message-----
> From: Jay Kreps [mailto:jay.kreps@gmail.com]
> Sent: Wednesday, March 05, 2014 10:10 PM
> To: dev@kafka.apache.org; users@kafka.apache.org
> Subject: Documentation for the upcoming 0.8.1 release
>
> Hey guys,
>
> I took a stab at updating the docs for the 0.8.1 release. In
> particular, I added a section on log compaction:
> http://kafka.apache.org/081/documentation.html#compaction
>
> I also updated the configs. This is all under 081, I will flip this
> over to be the main documentation when 0.8.1 is released but you can see
it now at
> http://kafka.apache.org/081/documentation.html#compaction
>
> Any comments on improvements we could make would be much appreciated!
>
> -Jay
>
>
Re: Documentation for the upcoming 0.8.1 release
Posted by Jay Kreps <ja...@gmail.com>.
I think I understand what you are saying. I think you are saying that
perhaps you could break up a given problem into a kind of "system state"
and the actual data rows? This is not something we have had a need to do...
-Jay
On Thu, Mar 6, 2014 at 12:26 AM, Vjeran Marcinko <
vjeran.marcinko@email.t-com.hr> wrote:
> Thanx Jay,
>
> Somewhat related to log compaction....
>
> Did you have a need at LinkedIn where application should publish 2 messages
> on separate topics for each action executed, one to "event data" topic
> demarking signalizing executed action, and another one to "state change"
> topic which ultimately is good for state replication and can be compacted
> after period of time? This state change messages should maybe even contain
> reference to event message to be able to see the cause of state change...
>
> Regards,
> Vjeran
>
> -----Original Message-----
> From: Jay Kreps [mailto:jay.kreps@gmail.com]
> Sent: Wednesday, March 05, 2014 10:10 PM
> To: dev@kafka.apache.org; users@kafka.apache.org
> Subject: Documentation for the upcoming 0.8.1 release
>
> Hey guys,
>
> I took a stab at updating the docs for the 0.8.1 release. In particular, I
> added a section on log compaction:
> http://kafka.apache.org/081/documentation.html#compaction
>
> I also updated the configs. This is all under 081, I will flip this over to
> be the main documentation when 0.8.1 is released but you can see it now at
> http://kafka.apache.org/081/documentation.html#compaction
>
> Any comments on improvements we could make would be much appreciated!
>
> -Jay
>
>
RE: Documentation for the upcoming 0.8.1 release
Posted by Vjeran Marcinko <vj...@email.t-com.hr>.
Thanx Jay,
Somewhat related to log compaction....
Did you have a need at LinkedIn where application should publish 2 messages
on separate topics for each action executed, one to "event data" topic
demarking signalizing executed action, and another one to "state change"
topic which ultimately is good for state replication and can be compacted
after period of time? This state change messages should maybe even contain
reference to event message to be able to see the cause of state change...
Regards,
Vjeran
-----Original Message-----
From: Jay Kreps [mailto:jay.kreps@gmail.com]
Sent: Wednesday, March 05, 2014 10:10 PM
To: dev@kafka.apache.org; users@kafka.apache.org
Subject: Documentation for the upcoming 0.8.1 release
Hey guys,
I took a stab at updating the docs for the 0.8.1 release. In particular, I
added a section on log compaction:
http://kafka.apache.org/081/documentation.html#compaction
I also updated the configs. This is all under 081, I will flip this over to
be the main documentation when 0.8.1 is released but you can see it now at
http://kafka.apache.org/081/documentation.html#compaction
Any comments on improvements we could make would be much appreciated!
-Jay