You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Simon Hørup Eskildsen <si...@sirupsen.com> on 2013/10/15 18:56:09 UTC
Flush configuration per topic
Hi Kafkas!
Reading through the documentation and code of Kafka, it seems there is no
feature to set flushing interval (messages/time) for a specific topic. I am
interested in this to get consistency for certain topics by flushing after
every message, while having eventual consistency for other topics (default).
Would there be interest in such feature? Then I might be curious to give
the Log module a dive and see if I can foster enough Scala to add this.
Thanks!
--
Simon
http://sirupsen.com/
Re: Flush configuration per topic
Posted by Simon Hørup Eskildsen <si...@sirupsen.com>.
That's only the time delay. There seems to be no option to persist on every
message.
On Wed, Oct 16, 2013 at 12:05 AM, Jun Rao <ju...@gmail.com> wrote:
> In 0.8, we do have "log.flush.interval.ms.per.topic" (see
> http://kafka.apache.org/documentation.html#brokerconfigs for details).
>
> Thanks,
>
> Jun
>
>
> On Tue, Oct 15, 2013 at 5:47 PM, Simon Hørup Eskildsen
> <si...@sirupsen.com>wrote:
>
> > Do you mean that it's possible to override log configurations per topic
> in
> > trunk?
> >
> > Yeah, you're right. :-) I wasn't sure what to call it if not consistency,
> > even though I know that sort of has another meaning in this context.
> >
> >
> > On Tue, Oct 15, 2013 at 6:53 PM, Jay Kreps <ja...@gmail.com> wrote:
> >
> > > Yeah, looks like you are right, we don't have the per-topic override in
> > 0.8
> > > :-(
> > >
> > > All log configurations are overridable in trunk which will be 0.8.1.
> > >
> > > Just to be totally clear this setting does not impact consistency (i.e.
> > all
> > > replicas will have the same messages in the same order), nor even
> > > durability (if you have replication > 1), but just recoverability on a
> > > single server in the event of a hard machine crash.
> > >
> > > -Jay
> > >
> > >
> > > On Tue, Oct 15, 2013 at 2:07 PM, Simon Hørup Eskildsen
> > > <si...@sirupsen.com>wrote:
> > >
> > > > 0.8, we're not on master, but we definitely can be.
> > > >
> > > >
> > > > On Tue, Oct 15, 2013 at 5:03 PM, Jay Kreps <ja...@gmail.com>
> > wrote:
> > > >
> > > > > Hey Simon,
> > > > >
> > > > > What version of Kafka are you using?
> > > > >
> > > > > -Jay
> > > > >
> > > > >
> > > > > On Tue, Oct 15, 2013 at 9:56 AM, Simon Hørup Eskildsen
> > > > > <si...@sirupsen.com>wrote:
> > > > >
> > > > > > Hi Kafkas!
> > > > > >
> > > > > > Reading through the documentation and code of Kafka, it seems
> there
> > > is
> > > > no
> > > > > > feature to set flushing interval (messages/time) for a specific
> > > topic.
> > > > I
> > > > > am
> > > > > > interested in this to get consistency for certain topics by
> > flushing
> > > > > after
> > > > > > every message, while having eventual consistency for other topics
> > > > > > (default).
> > > > > >
> > > > > > Would there be interest in such feature? Then I might be curious
> to
> > > > give
> > > > > > the Log module a dive and see if I can foster enough Scala to add
> > > this.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > --
> > > > > > Simon
> > > > > > http://sirupsen.com/
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Simon
> > > > http://sirupsen.com/
> > > >
> > >
> >
> >
> >
> > --
> > Simon
> > http://sirupsen.com/
> >
>
--
Simon
http://sirupsen.com/
Re: Flush configuration per topic
Posted by Jun Rao <ju...@gmail.com>.
In 0.8, we do have "log.flush.interval.ms.per.topic" (see
http://kafka.apache.org/documentation.html#brokerconfigs for details).
Thanks,
Jun
On Tue, Oct 15, 2013 at 5:47 PM, Simon Hørup Eskildsen
<si...@sirupsen.com>wrote:
> Do you mean that it's possible to override log configurations per topic in
> trunk?
>
> Yeah, you're right. :-) I wasn't sure what to call it if not consistency,
> even though I know that sort of has another meaning in this context.
>
>
> On Tue, Oct 15, 2013 at 6:53 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Yeah, looks like you are right, we don't have the per-topic override in
> 0.8
> > :-(
> >
> > All log configurations are overridable in trunk which will be 0.8.1.
> >
> > Just to be totally clear this setting does not impact consistency (i.e.
> all
> > replicas will have the same messages in the same order), nor even
> > durability (if you have replication > 1), but just recoverability on a
> > single server in the event of a hard machine crash.
> >
> > -Jay
> >
> >
> > On Tue, Oct 15, 2013 at 2:07 PM, Simon Hørup Eskildsen
> > <si...@sirupsen.com>wrote:
> >
> > > 0.8, we're not on master, but we definitely can be.
> > >
> > >
> > > On Tue, Oct 15, 2013 at 5:03 PM, Jay Kreps <ja...@gmail.com>
> wrote:
> > >
> > > > Hey Simon,
> > > >
> > > > What version of Kafka are you using?
> > > >
> > > > -Jay
> > > >
> > > >
> > > > On Tue, Oct 15, 2013 at 9:56 AM, Simon Hørup Eskildsen
> > > > <si...@sirupsen.com>wrote:
> > > >
> > > > > Hi Kafkas!
> > > > >
> > > > > Reading through the documentation and code of Kafka, it seems there
> > is
> > > no
> > > > > feature to set flushing interval (messages/time) for a specific
> > topic.
> > > I
> > > > am
> > > > > interested in this to get consistency for certain topics by
> flushing
> > > > after
> > > > > every message, while having eventual consistency for other topics
> > > > > (default).
> > > > >
> > > > > Would there be interest in such feature? Then I might be curious to
> > > give
> > > > > the Log module a dive and see if I can foster enough Scala to add
> > this.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > --
> > > > > Simon
> > > > > http://sirupsen.com/
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Simon
> > > http://sirupsen.com/
> > >
> >
>
>
>
> --
> Simon
> http://sirupsen.com/
>
Re: Flush configuration per topic
Posted by Felix GV <fe...@mate1inc.com>.
Ok cool, thanks Jun :)
--
Felix
On Fri, Oct 25, 2013 at 11:34 PM, Jun Rao <ju...@gmail.com> wrote:
> Yes, there won't be any feature development in 0.7.
>
> Thanks,
>
> Jun
>
>
> On Fri, Oct 25, 2013 at 6:32 PM, Felix GV <fe...@mate1inc.com> wrote:
>
> > Hi guys,
> >
> > I don't see log.flush.interval being available on per-topic basis in
> 0.7.2
> > either (only the time interval is configurable per topic), as in 0.8.
> >
> > Is it correct to say that this is only going to be available in 0.8.1 and
> > isn't available in 0.7.x?
> >
> > Thanks a lot guys (: ...
> >
> > --
> > Felix
> >
> >
> > On Fri, Oct 18, 2013 at 10:27 AM, Jay Kreps <ja...@gmail.com> wrote:
> >
> > > Both.
> > >
> > > On Thursday, October 17, 2013, Steve Morin wrote:
> > >
> > > > Is there only time delay or can it be set to flush for every message
> > with
> > > > the obvious performance hit?
> > > >
> > > >
> > > > On Wed, Oct 16, 2013 at 9:49 AM, Jay Kreps <jay.kreps@gmail.com
> > > <javascript:;>>
> > > > wrote:
> > > >
> > > > > Yes, the change in trunk is that all log configurations are
> > > automatically
> > > > > available at both the log level and the global default level and
> can
> > be
> > > > set
> > > > > at topic creation time or changed later without bouncing any
> servers.
> > > > >
> > > > > -Jay
> > > > >
> > > > >
> > > > > On Tue, Oct 15, 2013 at 5:47 PM, Simon Hørup Eskildsen
> > > > > <simon@sirupsen.com <javascript:;>>wrote:
> > > > >
> > > > > > Do you mean that it's possible to override log configurations per
> > > topic
> > > > > in
> > > > > > trunk?
> > > > > >
> > > > > > Yeah, you're right. :-) I wasn't sure what to call it if not
> > > > consistency,
> > > > > > even though I know that sort of has another meaning in this
> > context.
> > > > > >
> > > > > >
> > > > > > On Tue, Oct 15, 2013 at 6:53 PM, Jay Kreps <jay.kreps@gmail.com
> > > <javascript:;>>
> > > > wrote:
> > > > > >
> > > > > > > Yeah, looks like you are right, we don't have the per-topic
> > > override
> > > > in
> > > > > > 0.8
> > > > > > > :-(
> > > > > > >
> > > > > > > All log configurations are overridable in trunk which will be
> > > 0.8.1.
> > > > > > >
> > > > > > > Just to be totally clear this setting does not impact
> consistency
> > > > (i.e.
> > > > > > all
> > > > > > > replicas will have the same messages in the same order), nor
> even
> > > > > > > durability (if you have replication > 1), but just
> recoverability
> > > on
> > > > a
> > > > > > > single server in the event of a hard machine crash.
> > > > > > >
> > > > > > > -Jay
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Oct 15, 2013 at 2:07 PM, Simon Hørup Eskildsen
> > > > > > > <simon@sirupsen.com <javascript:;>>wrote:
> > > > > > >
> > > > > > > > 0.8, we're not on master, but we definitely can be.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Oct 15, 2013 at 5:03 PM, Jay Kreps <
> > jay.kreps@gmail.com
> > > <javascript:;>
> > > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Simon,
> > > > > > > > >
> > > > > > > > > What version of Kafka are you using?
> > > > > > > > >
> > > > > > > > > -Jay
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Oct 15, 2013 at 9:56 AM, Simon Hørup Eskildsen
> > > > > > > > > <simon@sirupsen.com <javascript:;>>wrote:
> > > > > > > > >
> > > > > > > > > > Hi Kafkas!
> > > > > > > > > >
> > > > > > > > > > Reading through the documentation and code of Kafka, it
> > seems
> > > > > there
> > > > > > > is
> > > > > > > > no
> > > > > > > > > > feature to set flushing interval (messages/time) for a
> > > specific
> > > > > > > topic.
> > > > > > > > I
> > > > > > > > > am
> > > > > > > > > > interested in this to get consistency for certain topics
> by
> > > > > > flushing
> > > > > > > > > after
> > > > > > > > > > every message, while having eventual consistency for
> other
> > > > topics
> > > > > > > > > > (default).
> > > > > > > > > >
> > > > > > > > > > Would there be interest in such feature? Then I might be
> > > > curious
> > > > > to
> > > > > > > > give
> > > > > > > > > > the Log module a dive and see if I can foster enough
> Scala
> > to
> > > > add
> > > > > > > this.
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Simon
> > > > > > > > > > http://sirupsen.com/
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Simon
> > > > > > > > http://sirupsen.com/
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Simon
> > > > > > http://sirupsen.com/
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Flush configuration per topic
Posted by Jun Rao <ju...@gmail.com>.
Yes, there won't be any feature development in 0.7.
Thanks,
Jun
On Fri, Oct 25, 2013 at 6:32 PM, Felix GV <fe...@mate1inc.com> wrote:
> Hi guys,
>
> I don't see log.flush.interval being available on per-topic basis in 0.7.2
> either (only the time interval is configurable per topic), as in 0.8.
>
> Is it correct to say that this is only going to be available in 0.8.1 and
> isn't available in 0.7.x?
>
> Thanks a lot guys (: ...
>
> --
> Felix
>
>
> On Fri, Oct 18, 2013 at 10:27 AM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Both.
> >
> > On Thursday, October 17, 2013, Steve Morin wrote:
> >
> > > Is there only time delay or can it be set to flush for every message
> with
> > > the obvious performance hit?
> > >
> > >
> > > On Wed, Oct 16, 2013 at 9:49 AM, Jay Kreps <jay.kreps@gmail.com
> > <javascript:;>>
> > > wrote:
> > >
> > > > Yes, the change in trunk is that all log configurations are
> > automatically
> > > > available at both the log level and the global default level and can
> be
> > > set
> > > > at topic creation time or changed later without bouncing any servers.
> > > >
> > > > -Jay
> > > >
> > > >
> > > > On Tue, Oct 15, 2013 at 5:47 PM, Simon Hørup Eskildsen
> > > > <simon@sirupsen.com <javascript:;>>wrote:
> > > >
> > > > > Do you mean that it's possible to override log configurations per
> > topic
> > > > in
> > > > > trunk?
> > > > >
> > > > > Yeah, you're right. :-) I wasn't sure what to call it if not
> > > consistency,
> > > > > even though I know that sort of has another meaning in this
> context.
> > > > >
> > > > >
> > > > > On Tue, Oct 15, 2013 at 6:53 PM, Jay Kreps <jay.kreps@gmail.com
> > <javascript:;>>
> > > wrote:
> > > > >
> > > > > > Yeah, looks like you are right, we don't have the per-topic
> > override
> > > in
> > > > > 0.8
> > > > > > :-(
> > > > > >
> > > > > > All log configurations are overridable in trunk which will be
> > 0.8.1.
> > > > > >
> > > > > > Just to be totally clear this setting does not impact consistency
> > > (i.e.
> > > > > all
> > > > > > replicas will have the same messages in the same order), nor even
> > > > > > durability (if you have replication > 1), but just recoverability
> > on
> > > a
> > > > > > single server in the event of a hard machine crash.
> > > > > >
> > > > > > -Jay
> > > > > >
> > > > > >
> > > > > > On Tue, Oct 15, 2013 at 2:07 PM, Simon Hørup Eskildsen
> > > > > > <simon@sirupsen.com <javascript:;>>wrote:
> > > > > >
> > > > > > > 0.8, we're not on master, but we definitely can be.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Oct 15, 2013 at 5:03 PM, Jay Kreps <
> jay.kreps@gmail.com
> > <javascript:;>
> > > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hey Simon,
> > > > > > > >
> > > > > > > > What version of Kafka are you using?
> > > > > > > >
> > > > > > > > -Jay
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Oct 15, 2013 at 9:56 AM, Simon Hørup Eskildsen
> > > > > > > > <simon@sirupsen.com <javascript:;>>wrote:
> > > > > > > >
> > > > > > > > > Hi Kafkas!
> > > > > > > > >
> > > > > > > > > Reading through the documentation and code of Kafka, it
> seems
> > > > there
> > > > > > is
> > > > > > > no
> > > > > > > > > feature to set flushing interval (messages/time) for a
> > specific
> > > > > > topic.
> > > > > > > I
> > > > > > > > am
> > > > > > > > > interested in this to get consistency for certain topics by
> > > > > flushing
> > > > > > > > after
> > > > > > > > > every message, while having eventual consistency for other
> > > topics
> > > > > > > > > (default).
> > > > > > > > >
> > > > > > > > > Would there be interest in such feature? Then I might be
> > > curious
> > > > to
> > > > > > > give
> > > > > > > > > the Log module a dive and see if I can foster enough Scala
> to
> > > add
> > > > > > this.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Simon
> > > > > > > > > http://sirupsen.com/
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Simon
> > > > > > > http://sirupsen.com/
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Simon
> > > > > http://sirupsen.com/
> > > > >
> > > >
> > >
> >
>
Re: Flush configuration per topic
Posted by Felix GV <fe...@mate1inc.com>.
Hi guys,
I don't see log.flush.interval being available on per-topic basis in 0.7.2
either (only the time interval is configurable per topic), as in 0.8.
Is it correct to say that this is only going to be available in 0.8.1 and
isn't available in 0.7.x?
Thanks a lot guys (: ...
--
Felix
On Fri, Oct 18, 2013 at 10:27 AM, Jay Kreps <ja...@gmail.com> wrote:
> Both.
>
> On Thursday, October 17, 2013, Steve Morin wrote:
>
> > Is there only time delay or can it be set to flush for every message with
> > the obvious performance hit?
> >
> >
> > On Wed, Oct 16, 2013 at 9:49 AM, Jay Kreps <jay.kreps@gmail.com
> <javascript:;>>
> > wrote:
> >
> > > Yes, the change in trunk is that all log configurations are
> automatically
> > > available at both the log level and the global default level and can be
> > set
> > > at topic creation time or changed later without bouncing any servers.
> > >
> > > -Jay
> > >
> > >
> > > On Tue, Oct 15, 2013 at 5:47 PM, Simon Hørup Eskildsen
> > > <simon@sirupsen.com <javascript:;>>wrote:
> > >
> > > > Do you mean that it's possible to override log configurations per
> topic
> > > in
> > > > trunk?
> > > >
> > > > Yeah, you're right. :-) I wasn't sure what to call it if not
> > consistency,
> > > > even though I know that sort of has another meaning in this context.
> > > >
> > > >
> > > > On Tue, Oct 15, 2013 at 6:53 PM, Jay Kreps <jay.kreps@gmail.com
> <javascript:;>>
> > wrote:
> > > >
> > > > > Yeah, looks like you are right, we don't have the per-topic
> override
> > in
> > > > 0.8
> > > > > :-(
> > > > >
> > > > > All log configurations are overridable in trunk which will be
> 0.8.1.
> > > > >
> > > > > Just to be totally clear this setting does not impact consistency
> > (i.e.
> > > > all
> > > > > replicas will have the same messages in the same order), nor even
> > > > > durability (if you have replication > 1), but just recoverability
> on
> > a
> > > > > single server in the event of a hard machine crash.
> > > > >
> > > > > -Jay
> > > > >
> > > > >
> > > > > On Tue, Oct 15, 2013 at 2:07 PM, Simon Hørup Eskildsen
> > > > > <simon@sirupsen.com <javascript:;>>wrote:
> > > > >
> > > > > > 0.8, we're not on master, but we definitely can be.
> > > > > >
> > > > > >
> > > > > > On Tue, Oct 15, 2013 at 5:03 PM, Jay Kreps <jay.kreps@gmail.com
> <javascript:;>
> > >
> > > > wrote:
> > > > > >
> > > > > > > Hey Simon,
> > > > > > >
> > > > > > > What version of Kafka are you using?
> > > > > > >
> > > > > > > -Jay
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Oct 15, 2013 at 9:56 AM, Simon Hørup Eskildsen
> > > > > > > <simon@sirupsen.com <javascript:;>>wrote:
> > > > > > >
> > > > > > > > Hi Kafkas!
> > > > > > > >
> > > > > > > > Reading through the documentation and code of Kafka, it seems
> > > there
> > > > > is
> > > > > > no
> > > > > > > > feature to set flushing interval (messages/time) for a
> specific
> > > > > topic.
> > > > > > I
> > > > > > > am
> > > > > > > > interested in this to get consistency for certain topics by
> > > > flushing
> > > > > > > after
> > > > > > > > every message, while having eventual consistency for other
> > topics
> > > > > > > > (default).
> > > > > > > >
> > > > > > > > Would there be interest in such feature? Then I might be
> > curious
> > > to
> > > > > > give
> > > > > > > > the Log module a dive and see if I can foster enough Scala to
> > add
> > > > > this.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > --
> > > > > > > > Simon
> > > > > > > > http://sirupsen.com/
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Simon
> > > > > > http://sirupsen.com/
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Simon
> > > > http://sirupsen.com/
> > > >
> > >
> >
>
Re: Flush configuration per topic
Posted by Jay Kreps <ja...@gmail.com>.
Both.
On Thursday, October 17, 2013, Steve Morin wrote:
> Is there only time delay or can it be set to flush for every message with
> the obvious performance hit?
>
>
> On Wed, Oct 16, 2013 at 9:49 AM, Jay Kreps <jay.kreps@gmail.com<javascript:;>>
> wrote:
>
> > Yes, the change in trunk is that all log configurations are automatically
> > available at both the log level and the global default level and can be
> set
> > at topic creation time or changed later without bouncing any servers.
> >
> > -Jay
> >
> >
> > On Tue, Oct 15, 2013 at 5:47 PM, Simon Hørup Eskildsen
> > <simon@sirupsen.com <javascript:;>>wrote:
> >
> > > Do you mean that it's possible to override log configurations per topic
> > in
> > > trunk?
> > >
> > > Yeah, you're right. :-) I wasn't sure what to call it if not
> consistency,
> > > even though I know that sort of has another meaning in this context.
> > >
> > >
> > > On Tue, Oct 15, 2013 at 6:53 PM, Jay Kreps <jay.kreps@gmail.com<javascript:;>>
> wrote:
> > >
> > > > Yeah, looks like you are right, we don't have the per-topic override
> in
> > > 0.8
> > > > :-(
> > > >
> > > > All log configurations are overridable in trunk which will be 0.8.1.
> > > >
> > > > Just to be totally clear this setting does not impact consistency
> (i.e.
> > > all
> > > > replicas will have the same messages in the same order), nor even
> > > > durability (if you have replication > 1), but just recoverability on
> a
> > > > single server in the event of a hard machine crash.
> > > >
> > > > -Jay
> > > >
> > > >
> > > > On Tue, Oct 15, 2013 at 2:07 PM, Simon Hørup Eskildsen
> > > > <simon@sirupsen.com <javascript:;>>wrote:
> > > >
> > > > > 0.8, we're not on master, but we definitely can be.
> > > > >
> > > > >
> > > > > On Tue, Oct 15, 2013 at 5:03 PM, Jay Kreps <jay.kreps@gmail.com<javascript:;>
> >
> > > wrote:
> > > > >
> > > > > > Hey Simon,
> > > > > >
> > > > > > What version of Kafka are you using?
> > > > > >
> > > > > > -Jay
> > > > > >
> > > > > >
> > > > > > On Tue, Oct 15, 2013 at 9:56 AM, Simon Hørup Eskildsen
> > > > > > <simon@sirupsen.com <javascript:;>>wrote:
> > > > > >
> > > > > > > Hi Kafkas!
> > > > > > >
> > > > > > > Reading through the documentation and code of Kafka, it seems
> > there
> > > > is
> > > > > no
> > > > > > > feature to set flushing interval (messages/time) for a specific
> > > > topic.
> > > > > I
> > > > > > am
> > > > > > > interested in this to get consistency for certain topics by
> > > flushing
> > > > > > after
> > > > > > > every message, while having eventual consistency for other
> topics
> > > > > > > (default).
> > > > > > >
> > > > > > > Would there be interest in such feature? Then I might be
> curious
> > to
> > > > > give
> > > > > > > the Log module a dive and see if I can foster enough Scala to
> add
> > > > this.
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > --
> > > > > > > Simon
> > > > > > > http://sirupsen.com/
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Simon
> > > > > http://sirupsen.com/
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Simon
> > > http://sirupsen.com/
> > >
> >
>
Re: Flush configuration per topic
Posted by Steve Morin <st...@stevemorin.com>.
Is there only time delay or can it be set to flush for every message with
the obvious performance hit?
On Wed, Oct 16, 2013 at 9:49 AM, Jay Kreps <ja...@gmail.com> wrote:
> Yes, the change in trunk is that all log configurations are automatically
> available at both the log level and the global default level and can be set
> at topic creation time or changed later without bouncing any servers.
>
> -Jay
>
>
> On Tue, Oct 15, 2013 at 5:47 PM, Simon Hørup Eskildsen
> <si...@sirupsen.com>wrote:
>
> > Do you mean that it's possible to override log configurations per topic
> in
> > trunk?
> >
> > Yeah, you're right. :-) I wasn't sure what to call it if not consistency,
> > even though I know that sort of has another meaning in this context.
> >
> >
> > On Tue, Oct 15, 2013 at 6:53 PM, Jay Kreps <ja...@gmail.com> wrote:
> >
> > > Yeah, looks like you are right, we don't have the per-topic override in
> > 0.8
> > > :-(
> > >
> > > All log configurations are overridable in trunk which will be 0.8.1.
> > >
> > > Just to be totally clear this setting does not impact consistency (i.e.
> > all
> > > replicas will have the same messages in the same order), nor even
> > > durability (if you have replication > 1), but just recoverability on a
> > > single server in the event of a hard machine crash.
> > >
> > > -Jay
> > >
> > >
> > > On Tue, Oct 15, 2013 at 2:07 PM, Simon Hørup Eskildsen
> > > <si...@sirupsen.com>wrote:
> > >
> > > > 0.8, we're not on master, but we definitely can be.
> > > >
> > > >
> > > > On Tue, Oct 15, 2013 at 5:03 PM, Jay Kreps <ja...@gmail.com>
> > wrote:
> > > >
> > > > > Hey Simon,
> > > > >
> > > > > What version of Kafka are you using?
> > > > >
> > > > > -Jay
> > > > >
> > > > >
> > > > > On Tue, Oct 15, 2013 at 9:56 AM, Simon Hørup Eskildsen
> > > > > <si...@sirupsen.com>wrote:
> > > > >
> > > > > > Hi Kafkas!
> > > > > >
> > > > > > Reading through the documentation and code of Kafka, it seems
> there
> > > is
> > > > no
> > > > > > feature to set flushing interval (messages/time) for a specific
> > > topic.
> > > > I
> > > > > am
> > > > > > interested in this to get consistency for certain topics by
> > flushing
> > > > > after
> > > > > > every message, while having eventual consistency for other topics
> > > > > > (default).
> > > > > >
> > > > > > Would there be interest in such feature? Then I might be curious
> to
> > > > give
> > > > > > the Log module a dive and see if I can foster enough Scala to add
> > > this.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > --
> > > > > > Simon
> > > > > > http://sirupsen.com/
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Simon
> > > > http://sirupsen.com/
> > > >
> > >
> >
> >
> >
> > --
> > Simon
> > http://sirupsen.com/
> >
>
Re: Flush configuration per topic
Posted by Jay Kreps <ja...@gmail.com>.
Yes, the change in trunk is that all log configurations are automatically
available at both the log level and the global default level and can be set
at topic creation time or changed later without bouncing any servers.
-Jay
On Tue, Oct 15, 2013 at 5:47 PM, Simon Hørup Eskildsen
<si...@sirupsen.com>wrote:
> Do you mean that it's possible to override log configurations per topic in
> trunk?
>
> Yeah, you're right. :-) I wasn't sure what to call it if not consistency,
> even though I know that sort of has another meaning in this context.
>
>
> On Tue, Oct 15, 2013 at 6:53 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Yeah, looks like you are right, we don't have the per-topic override in
> 0.8
> > :-(
> >
> > All log configurations are overridable in trunk which will be 0.8.1.
> >
> > Just to be totally clear this setting does not impact consistency (i.e.
> all
> > replicas will have the same messages in the same order), nor even
> > durability (if you have replication > 1), but just recoverability on a
> > single server in the event of a hard machine crash.
> >
> > -Jay
> >
> >
> > On Tue, Oct 15, 2013 at 2:07 PM, Simon Hørup Eskildsen
> > <si...@sirupsen.com>wrote:
> >
> > > 0.8, we're not on master, but we definitely can be.
> > >
> > >
> > > On Tue, Oct 15, 2013 at 5:03 PM, Jay Kreps <ja...@gmail.com>
> wrote:
> > >
> > > > Hey Simon,
> > > >
> > > > What version of Kafka are you using?
> > > >
> > > > -Jay
> > > >
> > > >
> > > > On Tue, Oct 15, 2013 at 9:56 AM, Simon Hørup Eskildsen
> > > > <si...@sirupsen.com>wrote:
> > > >
> > > > > Hi Kafkas!
> > > > >
> > > > > Reading through the documentation and code of Kafka, it seems there
> > is
> > > no
> > > > > feature to set flushing interval (messages/time) for a specific
> > topic.
> > > I
> > > > am
> > > > > interested in this to get consistency for certain topics by
> flushing
> > > > after
> > > > > every message, while having eventual consistency for other topics
> > > > > (default).
> > > > >
> > > > > Would there be interest in such feature? Then I might be curious to
> > > give
> > > > > the Log module a dive and see if I can foster enough Scala to add
> > this.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > --
> > > > > Simon
> > > > > http://sirupsen.com/
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Simon
> > > http://sirupsen.com/
> > >
> >
>
>
>
> --
> Simon
> http://sirupsen.com/
>
Re: Flush configuration per topic
Posted by Simon Hørup Eskildsen <si...@sirupsen.com>.
Do you mean that it's possible to override log configurations per topic in
trunk?
Yeah, you're right. :-) I wasn't sure what to call it if not consistency,
even though I know that sort of has another meaning in this context.
On Tue, Oct 15, 2013 at 6:53 PM, Jay Kreps <ja...@gmail.com> wrote:
> Yeah, looks like you are right, we don't have the per-topic override in 0.8
> :-(
>
> All log configurations are overridable in trunk which will be 0.8.1.
>
> Just to be totally clear this setting does not impact consistency (i.e. all
> replicas will have the same messages in the same order), nor even
> durability (if you have replication > 1), but just recoverability on a
> single server in the event of a hard machine crash.
>
> -Jay
>
>
> On Tue, Oct 15, 2013 at 2:07 PM, Simon Hørup Eskildsen
> <si...@sirupsen.com>wrote:
>
> > 0.8, we're not on master, but we definitely can be.
> >
> >
> > On Tue, Oct 15, 2013 at 5:03 PM, Jay Kreps <ja...@gmail.com> wrote:
> >
> > > Hey Simon,
> > >
> > > What version of Kafka are you using?
> > >
> > > -Jay
> > >
> > >
> > > On Tue, Oct 15, 2013 at 9:56 AM, Simon Hørup Eskildsen
> > > <si...@sirupsen.com>wrote:
> > >
> > > > Hi Kafkas!
> > > >
> > > > Reading through the documentation and code of Kafka, it seems there
> is
> > no
> > > > feature to set flushing interval (messages/time) for a specific
> topic.
> > I
> > > am
> > > > interested in this to get consistency for certain topics by flushing
> > > after
> > > > every message, while having eventual consistency for other topics
> > > > (default).
> > > >
> > > > Would there be interest in such feature? Then I might be curious to
> > give
> > > > the Log module a dive and see if I can foster enough Scala to add
> this.
> > > >
> > > > Thanks!
> > > >
> > > > --
> > > > Simon
> > > > http://sirupsen.com/
> > > >
> > >
> >
> >
> >
> > --
> > Simon
> > http://sirupsen.com/
> >
>
--
Simon
http://sirupsen.com/
Re: Flush configuration per topic
Posted by Jay Kreps <ja...@gmail.com>.
Yeah, looks like you are right, we don't have the per-topic override in 0.8
:-(
All log configurations are overridable in trunk which will be 0.8.1.
Just to be totally clear this setting does not impact consistency (i.e. all
replicas will have the same messages in the same order), nor even
durability (if you have replication > 1), but just recoverability on a
single server in the event of a hard machine crash.
-Jay
On Tue, Oct 15, 2013 at 2:07 PM, Simon Hørup Eskildsen
<si...@sirupsen.com>wrote:
> 0.8, we're not on master, but we definitely can be.
>
>
> On Tue, Oct 15, 2013 at 5:03 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Hey Simon,
> >
> > What version of Kafka are you using?
> >
> > -Jay
> >
> >
> > On Tue, Oct 15, 2013 at 9:56 AM, Simon Hørup Eskildsen
> > <si...@sirupsen.com>wrote:
> >
> > > Hi Kafkas!
> > >
> > > Reading through the documentation and code of Kafka, it seems there is
> no
> > > feature to set flushing interval (messages/time) for a specific topic.
> I
> > am
> > > interested in this to get consistency for certain topics by flushing
> > after
> > > every message, while having eventual consistency for other topics
> > > (default).
> > >
> > > Would there be interest in such feature? Then I might be curious to
> give
> > > the Log module a dive and see if I can foster enough Scala to add this.
> > >
> > > Thanks!
> > >
> > > --
> > > Simon
> > > http://sirupsen.com/
> > >
> >
>
>
>
> --
> Simon
> http://sirupsen.com/
>
Re: Flush configuration per topic
Posted by Simon Hørup Eskildsen <si...@sirupsen.com>.
0.8, we're not on master, but we definitely can be.
On Tue, Oct 15, 2013 at 5:03 PM, Jay Kreps <ja...@gmail.com> wrote:
> Hey Simon,
>
> What version of Kafka are you using?
>
> -Jay
>
>
> On Tue, Oct 15, 2013 at 9:56 AM, Simon Hørup Eskildsen
> <si...@sirupsen.com>wrote:
>
> > Hi Kafkas!
> >
> > Reading through the documentation and code of Kafka, it seems there is no
> > feature to set flushing interval (messages/time) for a specific topic. I
> am
> > interested in this to get consistency for certain topics by flushing
> after
> > every message, while having eventual consistency for other topics
> > (default).
> >
> > Would there be interest in such feature? Then I might be curious to give
> > the Log module a dive and see if I can foster enough Scala to add this.
> >
> > Thanks!
> >
> > --
> > Simon
> > http://sirupsen.com/
> >
>
--
Simon
http://sirupsen.com/
Re: Flush configuration per topic
Posted by Jay Kreps <ja...@gmail.com>.
Hey Simon,
What version of Kafka are you using?
-Jay
On Tue, Oct 15, 2013 at 9:56 AM, Simon Hørup Eskildsen
<si...@sirupsen.com>wrote:
> Hi Kafkas!
>
> Reading through the documentation and code of Kafka, it seems there is no
> feature to set flushing interval (messages/time) for a specific topic. I am
> interested in this to get consistency for certain topics by flushing after
> every message, while having eventual consistency for other topics
> (default).
>
> Would there be interest in such feature? Then I might be curious to give
> the Log module a dive and see if I can foster enough Scala to add this.
>
> Thanks!
>
> --
> Simon
> http://sirupsen.com/
>